Applications hosted on Windows Azure will create log files and given that we don’t have access to the running app instances the log files are the only means available to us for debugging and problem resolution so they’re a very important part of the whole service. Recently I wanted to take a look at the log files for my Tweetpoll Azure application and realised that viewing and analysing those log files is not exactly an easy process. Azure log files are stored in Azure BLOB storage from where you able to download them for offline analysis (N.B. I won’t be covering in this blog post the process of obtaining your log files – I’ll assume you already know how to do that).
Azure logs are delivered in XML files. Here’s one of them:
Given that I know quite a bit about SQL Server Integration Services (SSIS) I figured I would build a SSIS package that would parse my files for me; once parsed I can use SSIS to load the files to wherever I like. Perhaps I load the data into a database so I could query over it, perhaps I even push it into an Analysis Services cube so that I can slice-and-dice it; lots of possibilities.
First job though is to parse the files. I loop over them using a For Each loop container and upon each iteration push the log file contents into a raw file (SSIS’s proprietary storage format), appending the data a log file at a time. Once in the raw file I can consume all the log file data en masse and push it to wherever I want. TO visualise that for you, here is the control-flow of my package:
As I explained above, “DFT Consume all log files” does not actually push the data anywhere but I have hooked up some data viewers so you can see the data as it passes thru when the package is running in debug mode:
That’s all there is to it. If you’re familiar with SSIS you will know that the ‘Union All’ components shown here simply need to be replaced with OLE DB Destination components in order to push the data into a database table for analysis – what you do with the data thereafter is entirely up to you.
You can download the package from my Skydrive:
There is one simple piece of configuration you need to carry out before being able to run the package; you need to store the path to the folder containing your log files in the @[User::RootFolder] variable:
There are two pre-requisites to running the package. You need to have SQL Server Integration Services 2008 installed and you need to have read-write access to the folder containing your log files (write access is required because that is where the package attempt to write the raw file).
Hope that proves useful to people! Any questions please don’t hesitate to ask.
-Jamie
P.S. I recognise that the process of downloading all your files is laborious, I’m working on a more automated solution to that. Watch this space ;)