Logging with Log4Net on the Azure platform

Log4Net on Azure

I have always been a big fan of the Log4Net framework for all of my logging needs in applications and I would like to use this framework in my azure projects as well. And to be honest it isn’t even that difficult: just configure it to the TraceAppender as everything written to the tracelog will be transferred by the diagnostics manager to azure table storage. 

But there are two problems, or inconveniences, with this approach. First of all, you have to both configure log4net and the diagnostics monitor. This is a pretty repetitive task that you don’t want to perform over and over again for every application you create.  Secondly log4net is configured using the application’s configuration file, which is rather inconvenient on azure as it isn’t editable.

AzureAppender

To solve both of these problems we can instead create a custom appender that both sets up the diagnostics monitor as well as read the log4net configuration from the service configuration file instead of the application configuration file.

In order to do so, you have to inherit a class from AppenderSkeleton.

public sealed class AzureAppender : AppenderSkeleton

First thing that needs to be done is ensure that the configuration values are read from the service configuration file, if they are present. This is a bit clumbsy on azure as you cannot check for the presence of a configuration key, all you can do is act on the exception thrown when the key is not present. Make sure to set all values before the ActivateOptions method is called.

The following example shows you how to read the error level from config and apply it to the log4net environment.

private static string GetLevel()        
{            
       try            
       {                
           return RoleEnvironment.GetConfigurationSettingValue(LevelKey);            
       }           
       catch (Exception)            
       {                
            return "Error";            
       }        
}

 private void ConfigureThreshold()
 {
     var rootRepository = (Hierarchy) log4net.LogManager.GetRepository();
     Threshold = rootRepository.LevelMap[GetLevel()];
 }

The appender for this article supports the following configuration settings:

  • Diagnostics.ConnectionString Sets the connection string to be used when transferring the log entries to table storage
  • Diagnostics.Level Sets the threshold that log4net will use to filter the logs to output.
  • Diagnostics.Layout Defines the layout and content that log4net will use to create the log entries
  • Diagnostics.ScheduledTransferPeriod Specifies the interval, in minutes, that will be used by the diagnostics manager to transfer logs to azure table storage
  • Diagnostics.EventLogs Configures which of the event log sections will be transferred from the azure instance to azure table storage

When the options have been set and activated, the log4net environment has been completely configured to make proper use of our custom appender and we can start azure diagnostics monitor. Note that the diagnostics monitor also has a threshold that allows you to filter the logs written to storage. But as log4net is already filtering, we don’t need to do it here anymore so we set the filter to Verbose.

 private void ConfigureAzureDiagnostics()
{
    var traceListener = new DiagnosticMonitorTraceListener();
    Trace.Listeners.Add(traceListener);

    var dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();

    //set threshold to verbose, what gets logged is controled by the log4net level
    dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;

    ScheduleTransfer(dmc);

    ConfigureWindowsEventLogsToBeTransferred(dmc);

    DiagnosticMonitor.Start(ConnectionStringKey, dmc);
}

private void ScheduleTransfer(DiagnosticMonitorConfiguration dmc)
{
    var transferPeriod = TimeSpan.FromMinutes(ScheduledTransferPeriod);
    dmc.Logs.ScheduledTransferPeriod = transferPeriod;
    dmc.WindowsEventLog.ScheduledTransferPeriod = transferPeriod;
}

private static void ConfigureWindowsEventLogsToBeTransferred(DiagnosticMonitorConfiguration dmc)
{
    var eventLogs = GetEventLogs().Split(';');
    foreach (var log in eventLogs)
    {
         dmc.WindowsEventLog.DataSources.Add(log);
    }
}

That’s all there is to it basically, the only thing we need to do now is to apply the appender to the environment. This is done by creating an instance of the appender, configure it either in code or using the settings in the service configuration file, and finally configure the log4net environment.

var appender = new AzureAppender();
appender.ActivateOptions();
BasicConfigurator.Configure(appender);

You can find the source for this appender in the NServiceBus project and feel free to use it in your projects. I also want to give special credits to Andreas Ohlund for creating the first version of this appender.

About these ads

25 Responses to Logging with Log4Net on the Azure platform

  1. Craig says:

    This looks really neat. Can you point me to an example of how to use this, specifically the configuration settings? I downloaded and looked in all the NServicebus code but couldn’t find any.

  2. Craig says:

    Further to my last comment, this is what I have in the cfg. Not sure if it is close or way off.

  3. I’ve uploaded a sample here http://www.4shared.com/file/fS64_x94/AzureLog4netAppender.html

    Configuration is done in the service configuration file as this is the only file you can actually manage online in. Web.config is rather useless in this environment…

  4. Craig says:

    Thank you very much for this.

  5. Nate says:

    Goeleven/Craig, Can you repost sample?

  6. Marco says:

    Do you have an example how to configure to have the logs go to a particular DataTable or Blob?
    I’m running your sample code (thanks for putting this together) and I see the logs in the output trace in VS2010

    But it was not directly clear to me how I configure this to use the storage

    Thanks again,

    Marco

  7. Hi Marco,

    You cannot control the table where diagnostics is writing to, or the blobs where other diagnostics are written to for that matter.

    The only thing you can control is the account used to write to, which can be done using the Diagnostics.Connectionstring setting specified in your service configuration file.

  8. Harper Shelby says:

    Great work, and thanks for sharing. I suspect I’m doing something wrong, though, as every log entry I write is getting repeated a lot of times. I’m using your configuration process, and I’m creating an instance of the AzureAppender in the base class constructor of my MVC controllers and my WCF service classes. Is this causing the problem directly, or is there some configuration I need to alter?

    TIA,
    Harper

    • Hi Harper,

      The behavior you see is related to the fact that you create your appender when creating a new controller or service, both MVC and WCF create a new instance for each request by default. You should instead create your appender whenever a role starts.

      Kind regards,
      Yves

      • Harper Shelby says:

        I see. I was afraid it was something like that. Thanks for the information.

  9. Harper Shelby says:

    OK – one more question, and I hope it’s not too basic. Do I still need to have my standard log4net configuration invoked that sets up the EventLogAppender in order to get the data out? I’m having issues seeing some of the logs, and those are missing from the project that doesn’t have the standard log4net configuration section.

  10. Hi Harper,

    If you’re using the appender standalone, then the answer is ‘yes you need to configure it the traditional way’.

    If used in context of NServiceBus, you don’t have to do it, as NServiceBus does it for you when adding the Appender using .Log4Net<AzureAppender>

    Kind regards,
    Yves

  11. Pingback: Migrating a web application to Azure | These Days Labs

  12. Hi Yves,

    Just wanted to say thanks for sharing, your log4net appender implementation does a good job in our solutions as well!

    Grtz
    Lennart

  13. Danny says:

    Daft question, Where do the logs get written to? When debugging I can see the logs in the Compure Emulator UI screen. But when I deploy, I can’t see any in the Table stroage or Blob storage. I have a few event logs in the WADWindowsEventLogsTable, but nothing that looks like it has come form my logging code.

    The is my config

    In webrole.cs I have the following

    public override bool OnStart()
    {
    string wadConnectionString = “Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString”;
    CloudStorageAccount storageAccount = CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue(wadConnectionString));

    var appender = new Classes.AzureAppender();
    appender.ActivateOptions();
    BasicConfigurator.Configure(appender);
    logger.Info(“log4net configured – detected Azure hosted”);

    logger.Info(“Starting web role”);

    return base.OnStart();
    }

    The Azure appender is as per except the connection stringkey was chaged to match
    private const string ConnectionStringKey = “Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString”;

    Have I missed anything. I commented out my normal log4net section in the web.config. I assumed this is correct as teh appender is being set up in code?

    Thanks in advance.

  14. Danny, as far as I can tell from your code, you’re not passing in the cloud storage account into the azure appender therefore it does not write to table storage but only to the console (thats why you see it at debug time)

    Which version of the appender are you using? Depending on that you need to either pass the account into the appender or configure a different config property called Diagnostics.ConnectionString (as used to be the default generated by visual studio about half a year ago)

    Kind regards,
    Yves

  15. Thank you for the great sample, this got me jumpstarted.

    I was wondering if this azure appender implementation works for you when you run the appender in from web and worker roles that have multiple instances?

    In my scenario, I have 2 web roles, with 2 instances each, and 1 worker role with 2 instances. All of my azure appenders seemed to have magically stopped working.

    All of my diagnostics are being output, except the diagnostics logs. My implementation of the azure appender was working but I cant figure out what I did to break it.

    I was wondering if you or anyone else is seeing odd behavior from time to time.

  16. Cedric,

    It all seems to work fine here, but I do have an updated version of this appender that now uses the .wadcfg diagnostics file instead… I’ll see if I can post about it in the next few weeks.

    Kind regards,
    Yves

  17. Pingback: Logger dans Azure (Log4Net, …) « Merroun

  18. cskr says:

    Hi Yves

    Can you please post the updated version.

  19. cskr says:

    We are moving our application to cloud and go struck with the configuration settings.

  20. Hi cskr,

    Here is the version I’m currently using: https://github.com/NServiceBus/NServiceBus/blob/master/src/azure/Integration/NServiceBus.Integration.Azure/AzureAppender.cs

    The only thing you have to do is add your connectionstring to the service configuration file and optionally set the warning level, like this:

    All other config stuff is best managed through the windows azure diagnostics configuration file, diagnostics.wadcfg, that you add as content file in the root of your role: For more information on the schema of this file see http://msdn.microsoft.com/en-us/library/windowsazure/ee758710.aspx

    That should be it… in case it doesn’t work out for you, send me a mail at yves@goeleven.com and I’ll try to help you.

    Kind regards,
    Yves

  21. Pingback: Logging in Azure: Part 2–Table Storage « Tyler's Developer Blog

  22. Pingback: Logging in Azure: Part 2–Table Storage | Tyler's Azure Developer Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: