Collecting Event Logs in Windows Azure
While playing with Windows Azure Diagnostics for my last post Windows Azure Diagnostics – Where Are My Logs? I noticed few things related to collecting Event Logs.
Configuring Windows Event Logs Data Source
First let’s configure the Event Logs data source. Here are the four simple lines that do that:
DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();
dmc.WindowsEventLog.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
I would highly recommend you to define the
“Application!*” string globally, and make sure it is not misspelled. I spent quite some time wondering why my Event Logs are not showing up in my storage table, and the reason was my bad spelling skills. BIG THANKS to Steve Marx for opening my eyes!
If you want to see other event logs like System and Security you should add those as data sources before you call
The first thing I noticed is that the configuration above will transfer everything from the Application Event Log to WADWindowsEventLogsTable at one minute intervals. With “everything” I literally mean everything – doesn’t matter whether the event is generated by your application or something else running in parallel. There are two reasons why you don’t want to dump everything into your storage table: 1. it is too much noise, and 2. (but more important) the more data you dump the higher bill you will get (you will be charged for transactions and for data stored).
My suggestion is to always capture filtered events, and transfer only those to the storage table. Steve has a very short but useful post explaining how to capture filtered Windows Events with Windows Azure Diagnostics using XPath expressions. In my case I filtered to only events generated by my Web Role.
How are Windows Event Logs Transferred to Storage?
Once I had events transferred to WADWindowsEventLogsTable in my storage account, I used Cerebrata’s Azure Diagnostics Manager to browse through the data. I was running my tests on Development Fabric and storing the data in Development Storage, and surprisingly for me I saw the events duplicated in WADWindowsEventLogsTable. My first thought was: “This is a bug!”, and then the second: “Will I be double-charged?” Nothing to worry about! (And again thanks to Steve for opening my eyes).
The explanation is simple. Because every Role Instance runs in separate VM, it has its own Windows Azure Monitoring Agent running. The Monitoring Agent for particular Role Instance reads the events from the Event Log for this particular VM, and transfers it to WADWindowsEventLogsTable. Thus you are able to see the events for every Role Instance. If you look at the WADWindowsEventLogsTable schema you will see that there is a column RoleInstance that identifies the Role Instance from which this event came. What happens in Development Fabric is that there is only one Event Log to read from, and because I had two instances running I was seeing the events duplicated.
Guidelines for Capturing Windows Events Using Windows Azure Diagnostics
As a conclusion here are some guidelines for capturing Windows Event Logs using Windows Azure Diagnostics.
- As you can read from my previous post Windows Azure Diagnostics – Where Are My Logs? by default no logs are transferred to Windows Azure Storage tables. You need to explicitly set
DiagnosticsMonitorconfiguration.WindowsEventLog.DataSourcesif you want to receive events into WADWindowsEventLogsTable
- Always capture filtered Windows Events using XPath expressions in WindowsEventLog.DataSources in order to avoid noise and unnecessary charges
- Keep in mind that Event Logs are collected in distributed environment, and you will need to mind the data you receive in your table. Using tools like Cerebrata’s Azure Diagnostics Managercan help here
Using Windows Azure Diagnostics is a good way to debug your application in the cloud, however you need to be careful not to grow your bill unnecessary.
Update: Two updates based on feedback that I received in the last few days:
- Although I mention above that you can configure Security as event source you should be aware that this works only in Windows Azure Development Fabric. You will not be able to collect Security events in Windows Azure cloud environment because your role has no admin privileges.
- I received the suggestion (guess from whom) to post the Xpath expression I use to filter the events to only my role. Here it is: