Building Global Web Applications With the Windows Azure Platform – Understanding capacity

In this second installment of the ‘Building Global Web Applications series’, I would like to discuss the concept of ‘Capacity’ as I feel that only few people understand that it is the secret of the utility model, the business model behind cloud computing.

I hear, and tell, very often that cloud computing is about ‘pay for use’. But only for a few resources this is actually completely true, for many it means ‘pay for what you could potentially use’, aka the capacity of a certain resource. Let’s have a look at the pricing table of windows azure compute instances as an example:

Compute Instance Size CPU Memory Instance Storage I/O Performance Cost per hour
Extra Small 1.0 GHz 768 MB 20 GB Low (5 Mbps) $0.05
Small 1.6 GHz 1.75 GB 225 GB Moderate (100 Mbps) $0.12
Medium 2 x 1.6 GHz 3.5 GB 490 GB High (200 Mbps) $0.24
Large 4 x 1.6 GHz 7 GB 1,000 GB High (400 Mbps) $0.48
Extra Large 8 x 1.6 GHz 14 GB 2,040 GB High (800 Mbps) $0.96

When you look at this table, you can see that every windows azure role has a ‘capacity’ in terms of cpu, memory, local disk space and I/O (which actually means bandwidth), in other words the extra small instance has a potential to perform roughly 1 billion instructions per second, store 768 MB of data in memory, cache 20 GB of data on disk and transfer 5 Megabits of data per second.

When serving web pages, your role will start showing a decline in performance when either one of these 4 capacities is completely utilised. When this happens you might be tempted to either scale up or scale out in order to increase the number of users you can handle, but to be honest, this might not be the best idea, because at the same time you’re also wasting part of the 3 other capacities of your instance.

Last time, I showed you a load test on a single extra small instance, that showed signs of running out of capacity when there were more than 30 concurrent users on it. But when monitoring the instance I noticed that neither, memory, cpu nor local disk space were a problem. Only 10% of the cpu was utilitised, 82% of the memory was utilised but most of this was by the OS itself and there was an abundance of free disk space. So the bottle neck must have been the bandwith…

Let’s analyse a request and see whether or not this is true, luckily loadimpact also has a page analyser that shows you which parts of a page take how much time… as you can see from the results below, most of the time is spent on waiting for the first byte of several images (which is represented by the green bar) and waiting for the download of the larger image (represented by the blue bar). All clear indicators of the low i/o performance of an extra small role.

Now in order to increase the utilisation of other capacity types in our role, as well as increase the number of users we can handle, we should remove this bottleneck.

Ofloading the static images, that don’t require computation or memory anyway, to another medium such as blob storage or the CDN is one of the prime options.  This allows the machine to handle more requests for dynamic pages and thus increases the utilisation of both cpu and memory.

Next time we will see what exactly the impact is of offloading images to either blob storage or the CDN and how this compares to scaling out…

Building Global Web Applications With the Windows Azure Platform – Introduction

I don’t know if you noticed, probably not, but I’ve put some content again on http://www.goeleven.com. This content will serve as a starting point to a new series that I’m writing. In this series I will discuss, step by step, what it takes to build global, highly scalable, highly available, high density and cheap web applications with the windows azure platform.

In this first stage I’ve just built a simple web application, using asp.net MVC, with some fairly static content: a razor layout page, a static content body, a css file, some images, nothing fancy… All of this is configured in an extra small webrole uploaded to http://goeleven-eu.cloudapp.net with only 1 instance. (http://www.goeleven.com is mapped to this address using a CNAME record at my DNS provider).

The general idea behind this series is to build on top of this basic sample, with more functionality and more windows azure features, and try out how the application will behave in the real world, in terms of performance, scalability, availability and so on. In order to achieve this we need to be able to simulate some real life load on our application, so I signed up at http://loadimpact.com which allows me to setup load tests with up to 5000 simulated users.

In a very first test I will ramp up to 50 concurrent users and see if this miniature application can handle it. 50 concurrent users means about a 1000 visits per hour (given that the average stay time is about 3 minutes), or 24000 visitors per day, this should definitly do for my simple site at this stage…

Note: If you want to derive the average number of concurrent users currently on your site, you can use the following formula: concurrent_users = (hourly_visits * time_on_site_in_seconds) / 3600

Now let’s have a look at the results:

1 Extra Small Instance - 50 Concurrent users

As you can see, the site survived the onslaught, albeit barely. There is a significant decline in performance when the number of concurrent users increases over 30, and an almost 300% increase in response time once we reach 50 concurrent users. I’m quite sure the site would actually break down if we increased the numbers only a bit more.

Breaking down is a subjective concept on the web, it does not mean that the webserver actually crashes, it means that the users go away from the page they were intending to visit. This graph shows the average load times for all of the users. Average really means that 50% of the requests took more than the amount of time displayed in the graph. Personally I consider a site broken if it’s requests take more than 3 seconds to load on average, which means 50% of it’s users had to wait more than 3 seconds before they got a response (which they won’t do anymore).

So what can we do to handle this problem? How can we serve more people? We either scale up or scale out, right?

If this is your first reaction, you didn’t get the utility model yet…

Utilities, like windows azure, are all about optimal use of capacity. Windows azure roles offer an array of different kinds of capacity (compute, memory, bandwidth) and I bet that not all of these are optimally used yet and only one of them is the bottleneck…

Next time we will look into this capacity topic a bit further and see how we can get some more juice out of this instance without having to pay a lot more…

NServiceBus on Sql Azure, sometimes stuff just works

It just works

Microsoft has put a tremendous amount of effort into Sql Azure to ensure that it works the same way as Sql Server on premises does. The fruits of all this labor are now reaped by us, in order to persist NServiceBus data, such as subscriptions and saga’s for example, on Sql Azure all you got to do is change your connection string.

To show this off, I’ve included a version of the Starbucks sample in the trunk of NServiceBus which has been configured to store it’s information on Sql Azure. You may recall this sample from my previous article so I’m not going to explain the scenario again, I’ll just highlight the differences.

Configuring the sample

In order to configure NServiceBus for Sql Azure, you have to configure it as you would for Sql Server using the DBSubscriptionStorage and the NHibernateSagaPersister configuration settings for respectively subscriptions and saga’s:

Configure.With()
    .Log4Net()
    .StructureMapBuilder(ObjectFactory.Container)

    .AzureConfigurationSource()
    .AzureMessageQueue().JsonSerializer()
    .DBSubcriptionStorage()
    .Sagas().NHibernateSagaPersister().NHibernateUnitOfWork()

    .UnicastBus()
    .LoadMessageHandlers()
    .IsTransactional(true)
    .CreateBus()
    .Start();

In your application configuration file you also need to include the NHibernate connection details using the DBSubscriptionStorageConfig and NHibernateSagaPersisterConfig configuration sections.

<section name="DBSubscriptionStorageConfig"
         type="NServiceBus.Config.DBSubscriptionStorageConfig, NServiceBus.Core" />
<section name="NHibernateSagaPersisterConfig"
         type="NServiceBus.Config.NHibernateSagaPersisterConfig, NServiceBus.Core" />

Furthermore you need to provide the details for your NHibernate connection, like the connection provider, the driver, the sql dialect and finally your connection string. Be sure to format your connection string using the sql azure recognized format and naming conventions.

<DBSubscriptionStorageConfig>
  <NHibernateProperties>
    <add Key="connection.provider"
          Value="NHibernate.Connection.DriverConnectionProvider"/>
    <add Key="connection.driver_class"
          Value="NHibernate.Driver.SqlClientDriver"/>
    <add Key="connection.connection_string"
          Value="Server=tcp:[yourserver].database.windows.net;Database=NServiceBus;User ID=[accountname]@[yourserver];Password=[accountpassword];Trusted_Connection=False;Encrypt=True;"/>
    <add Key="dialect"
          Value="NHibernate.Dialect.MsSql2005Dialect"/>
  </NHibernateProperties>
</DBSubscriptionStorageConfig>

<NHibernateSagaPersisterConfig>
  <NHibernateProperties>
    <add Key="connection.provider"
          Value="NHibernate.Connection.DriverConnectionProvider"/>
    <add Key="connection.driver_class"
          Value="NHibernate.Driver.SqlClientDriver"/>
    <add Key="connection.connection_string"
          Value="Server=tcp:[yourserver].database.windows.net;Database=NServiceBus;User ID=[accountname]@[yourserver];Password=[accountpassword];Trusted_Connection=False;Encrypt=True;"/>
    <add Key="dialect"
          Value="NHibernate.Dialect.MsSql2005Dialect"/>
  </NHibernateProperties>
</NHibernateSagaPersisterConfig>

Alright, that was it, validate your other settings, like the azure storage account used as a transport mechanism and hit F5, it just works!

The Saga pattern on Azure with NServiceBus

The Saga Pattern

As you probably know by now, I’ve been adding support for Windows Azure to the NServiceBus framework, mostly because I believe NServiceBus has the most comprehensive set of communication pattern implementations available for doing development on the Windows Azure Platform. And one of these is the Saga pattern! Before I show you how easy it is to get Saga’s working on Windows Azure, let’s take some time to discuss the pattern itself as odds are you never heard of it before.

In essence a Saga is an object that keeps track of the state of a conversation between an object and it’s different partners. By combining multiple saga’s, representing each partner in the conversation you can fulfill the role of an orchestration that overcomes some of the challenges faced by workflow implementations in a cloud environment such as Windows Azure.

Indeed, the saga is a very viable alternative to a workflow. I’m not going into a debate on which style is better, I feel they both deserve a spot in my toolcase and can perfectly be used in conjunction in a solution. But the most important difference though is that Saga’s are more responsive in nature, they will act on incoming messages and rarely be in charge of the entire conversation. Because of this responsive and typical asynchronous nature of the communication between saga’s, they are not dependent on a distributed transaction. This is quite usefull in the cloud as distributed transactions simply don’t exist there.

Configuring saga support in NServiceBus

Alright, now how easy is it to persist saga’s to Azure’s Table Storage? As anything in NServiceBus, you just configure it by calling a few methods at configuration time. The Sagas method enables saga support, and specifying that you want to persist it to Azure Table Storage is done by means of the AzureSagaPersister method.

Configure.With()
                .Log4Net()
                .StructureMapBuilder(ObjectFactory.Container)

                .AzureConfigurationSource()
                .AzureMessageQueue().JsonSerializer()
                .AzureSubcriptionStorage()
                .Sagas().AzureSagaPersister().NHibernateUnitOfWork()

                .UnicastBus()
                     .LoadMessageHandlers()
                     .IsTransactional(true)
                .CreateBus()
                .Start();

Of course you also need to specify a connection string representing your storage account in the web.config (or service definition file if hosted in azure). Another option that you can enable is schema creation, which is recommended for azure table storage so that it is ensured that all table names exist (table storage doesn’t have any other schema information).

<AzureSagaPersisterConfig ConnectionString="UseDevelopmentStorage=true" CreateSchema="true" />

The Starbucks sample

If you want to see saga’s in action on azure, you can have a look at a azurified version of the Starbucks example. If you’ve never heard of the Starbucks example, it was originally defined by ayende on his blog and has become the default example for demonstrating the use of saga’s to implement a orchestration between humans and services. The implementation can be found in the NServiceBus trunk.

The scenario goes as follows:

I want to order a venti hot chocolate:

  • I (client) ask the cashier (external facing service) for a venti hot chocolate.
  • The cashier informs the barista (internal service) that a new order for venti hot chocolate has arrived.
  • The barista starts preparing the drink.
  • The cashier starts the payment process (credit card auth, or just counting change).
  • When the cashier finishes the payment process, he tells the barista that the payment process is done.
  • I (client) move to the outgoing coffee stand, and wait for my coffee.
  • The barista finishes preparing the drink, check that the payment for this order was done, and serve the drink.
  • I pick up my drink, and the whole thing is finished.

The implementation of the workflow consists of a saga for each the Cashier and the Barista. There is no saga for the customer as we humans tend to perform our communication orchestration ourselves. I’ll show you the code of the Cashier saga to explain how a saga is implemented, you can find the full source in the NServiceBus trunk of course.

First of all, we need to give our saga a memory, a data structure used to contain the state of the conversation. This data structure must implement the IContainSagaData interface to fulfill the identification requirements such as an Id and a representation of the originator. Furthermore we need to add some properties representing the type of drink, the amount ordered and the name of the customer to get the drink back to him or her.

public class CashierSagaData : IContainSagaData
{
    public virtual Guid Id { get; set; }
    public virtual String Originator { get; set; }
    public virtual String OriginalMessageId { get; set; }

    public virtual Double Amount { get; set; }
    public virtual String CustomerName { get; set; }
    public virtual String Drink { get; set; }
    public virtual DrinkSize DrinkSize { get; set; }
    public virtual Guid OrderId { get; set; }
}

The implementation of the saga itself requires you specify the state as the closing of the Saga generic type used as a base implementation. Furthermore you need to specify for which messages the saga will take part in the conversation. One specific message will mark the start of the conversation, in this example a new order, which is indicated by the IAmStartedByMessages interface. Other messages are handled with the traditional IHandleMessages interface.

public class CashierSaga : Saga<CashierSagaData>,
                            IAmStartedByMessages<NewOrderMessage>,
                            IHandleMessages<PaymentMessage>
{
    private readonly IStarbucksCashierView _view;

    public CashierSaga()
    {}

    public CashierSaga(IStarbucksCashierView view)
    {
        _view = view;
    }

We also need to specify when a saga instance should be loaded from persistence and play part in the conversation. Many orchestration frameworks would require you to pass the orchestration ID along with all of the messages, but NServiceBus has opted to implement this the other way around, we will map existing message information to the saga’s identifier using the ConfigureHowToFindSaga and ConfigureMapping methods.

    public override void ConfigureHowToFindSaga()
    {
        ConfigureMapping(s => s.OrderId, m => m.OrderId);
        ConfigureMapping(s => s.OrderId, m => m.OrderId);
    }

Defining the orchestration of the conversation is done by means of the individual message handlers, updating the state of the saga’s data and publishing, or replying to, other messages through the bus. The end of the conversation is marked by the MarkAsComplete method, this will in fact remove the saga from the conversation. The code below shows you how the cashier communicates with both the customer and the barista.

        
    public void Handle(NewOrderMessage message)
    {
        _view.NewOrder(new NewOrderView(message));

        Data.Drink = message.Drink;
        Data.DrinkSize = message.DrinkSize;
        Data.OrderId = message.OrderId;
        Data.CustomerName = message.CustomerName;
        Data.Amount = CalculateAmountAccordingTo(message.DrinkSize);

        Bus.Publish(new PrepareOrderMessage(Data.CustomerName, Data.Drink, Data.DrinkSize, Data.OrderId));
        Bus.Reply(new PaymentRequestMessage(Data.Amount, message.CustomerName, message.OrderId));
    }

    public void Handle(PaymentMessage message)
    {
        if(message.Amount >= Data.Amount)
        {
            var viewData = new ReceivedFullPaymentView(Data.CustomerName, Data.Drink, Data.DrinkSize);
            _view.ReceivedFullPayment(viewData);

            Bus.Publish(new PaymentCompleteMessage(Data.OrderId));
        }
        else if(message.Amount == 0)
        {
            var viewData = new CustomerRefusesToPayView(Data.CustomerName, Data.Amount, Data.Drink, Data.DrinkSize);
            _view.CustomerRefusesToPay(viewData);
        }

        MarkAsComplete();
    }

    private static Double CalculateAmountAccordingTo(DrinkSize size)
    {
        switch(size)
        {
            case DrinkSize.Tall:
                return 3.25;
            case DrinkSize.Grande:
                return 4.00;
            case DrinkSize.Venti:
                return 4.75;
            default:
                throw new InvalidOperationException(String.Format("Size '{0}' does not compute!", size));
        }
    }
}

Hope this example shows you how you can implement saga’s, as an alternative to workflow orchestrations, on the windows azure platform. But before I leave you, there is still one thing to remember:

One thing to remember

This implementation uses Azure Table storage under the hood and Azure table storage only supports a small number of data types as specified in following msdn article. It speaks for itself that you can only use compatible .Net types in your saga’s state. If you need more data types, you will have to consider SQL Azure as your data store, which I will show you how to do in a next post…

Publish/Subscribe on Azure with NServiceBus

The Pattern

The Pub/Sub (or Publish/Subscribe) pattern is a loosely coupled messaging pattern between different components in a system. Certain components, the subscribers, show their interest in certain messages by adding their endpoint to a list of subscriptions for those messages. The publishers will consult this list of subscriptions and send the message to every endpoint subscribed to the message.

This pattern is one of the many messaging patterns that are supported by the NServiceBus framework, and of course I added support to it for using this pattern on Azure as well, using table storage or sql azure. If you want more information on how NServiceBus supports Pub/Sub read the following article: http://www.nservicebus.com/PubSub.aspx

Configuring NServiceBus for Pub/Sub on Azure

Setting up NServiceBus to store the subscription list on azure table storage can be done by simply adding a configuration setting to the role’s configuration file that defines the connection string. This connection string has to follow the format defined by Microsoft at http://msdn.microsoft.com/en-us/library/ee758697.aspx.

<Setting name="AzureSubscriptionStorageConfig.ConnectionString"
        value="UseDevelopmentStorage=true" />

Besides adding the connection string, you also need to call the AzureSubscriptionStorage configuration method when configuring your bus.

Bus = Configure.With()
            .DefaultBuilder()
            .Log4Net(new AzureAppender())
            .AzureConfigurationSource()
            .AzureMessageQueue()
                .JsonSerializer()
            .AzureSubcriptionStorage()
            .UnicastBus()
                .LoadMessageHandlers()
                .IsTransactional(true)
            .CreateBus()
            .Start();

Configuring the Azure Host

As an alternative to manually configuring the bus in a worker role, you can also use the RoleEntryPoint defined in the NServiceBus.Hosting.Azure assembly. You need to inherit from this class so that the azure startup conventions can detect your entrypoint.

public class Host : RoleEntryPoint{}

The host takes a number of marker interfaces into account when starting the bus, such as AsA_Worker for example. This marker interface will ensure the setup of the azure subscription storage as well as saga storage, just don’t forget to include the configuration setting in you’re roles configuration file otherwise it will default to using the development storage emulator.

public class EndpointConfiguration : IConfigureThisEndpoint, AsA_Worker { }

As mentioned, we support more than 1 store for state information azure table storage and sql azure. This can be controlled by using profiles, in this case I’ve set the profile to

NServiceBus.Production NServiceBus.OnAzureTableStorage NServiceBus.WithAzureStorageQueues

The Pub/Sub Sample

The NServiceBus sources contain a sample that demonstrates the pub/sub messaging pattern on Windows Azure. It contains a web role that submits an order to a worker role for approval.

protected void btnSubmit_Click(object sender, EventArgs e)
{
    var order = new Order
                    {
                        Id = Guid.NewGuid(),
                        Quantity = Convert.ToInt32(txtQuatity.Text),
                        Status = OrderStatus.Pending

                    };

    Global.Bus.Send(new OrderMessage
                        {
                            Id = order.Id,
                            Quantity = order.Quantity
                        });

    lock (Global.Orders)
        Global.Orders.Add(order);

    Refresh();
}

Once the order has been submitted it will listen for the approval (or denial) by subscribing to the OrderUpdatedEvent, which is handled by the following message handler for the event.

public class OrderUpdatedEventHandler : IHandleMessages<OrderUpdatedEvent>
{
    public void Handle(OrderUpdatedEvent message)
    {
        var order = message.UpdatedOrder;

        lock (Global.Orders)
        {
            if (Global.Orders.Contains(order))
                Global.Orders.Remove(order);

            Global.Orders.Add(order);
        }
    }
}

The worker role will handle the submission of the order, approve it, and publish the order updated event. Using the Publish method on the bus will make sure that the message is send to every subscriber and not just to one of the web roles.

public void Handle(OrderMessage message)
{
    var order = new Order
                    {
                        Id = message.Id,
                        Quantity = message.Quantity
                    };

    //simlute business logic
    order.Status = message.Quantity < 100 ? OrderStatus.Approved : OrderStatus.AwaitingApproval;

    //publish update
    var orderUpdatedEvent = bus.CreateInstance(x=>x.UpdatedOrder = order);
    bus.Publish(orderUpdatedEvent);
}

Now it’s up to you…

Pub/Sub is probably one of the most important messaging patterns for doing development on windows azure, as it allows you to send events to all instances of a web or worker role, enabling really interesting use cases (from cache invalidation to complete event driven architectures). Therefore I advise you to get the latest version of NServiceBus from github, https://github.com/NServiceBus/NServiceBus, and get started with the Pub/Sub sample.

Windows Azure MVP

Quite unexpectedly, when browsing my mail over the weekend, I learned that I’ve been awarded as MVP for Windows Azure by Microsoft 🙂 This recognition of all my community work makes me extremely happy, thanks to all who have voted for me!

Operational costs of an Azure Message Queue

A potential issue with azure message queues is the fact that there are hidden costs associated with them, depending on how often the system accesses the queues you pay more, no matter whether there are messages or not.

Lets have a look at what the operational cost model for NServiceBus would look like if we treated azure queues the same as we do msmq queues.

On azure you pay $0.01 per 10K storage transactions. Every message send is a transaction, every successful message read are 2 transactions (GET + DELETE) and also every poll without a resulting message is a transaction. Multiply this by the number of roles you have, and again by the number of threads in the role and again by the number of polls the cpu can launch in an infinite for-loop per second and you’ve got yourself quite a bill at the end of the month. For example, 2 roles with 4 threads each, in idle state, at a  rate of 100 polls per second would result in an additional $2.88 per hour for accessing your queues.

Scary isn’t it 🙂

As a modern day developer you have the additional responsibility to take these operational costs into account and balance them against other requirements such as performance. In order for you to achieve the correct balance, I’ve implemented a configurable back-off mechanism in the NServiceBus implementation.

The basic idea is, if the thread has just processed a message it’s very likely that there are more messages, so we check the queue again in order to maintain high throughput. But if there was no message, we will delay our next read a bit before checking again. If there is still no message, we delay the next read a little more, and so on and so on, until a certain threshold has been reached. This poll interval will be maintained until there is a new message on the queue. Once a message has been processed we start the entire sequence again from zero…

Configuring the increment of the poll interval can be done by setting the PollIntervalIncrement property (in milliseconds) on the AzureQueueConfig configuration section, by default the interval is increased by 50 milliseconds at a time. To define the maximum wait time when there are no messages, you can configure the MaximumWaitTimeWhenIdle property (also in milliseconds). By default this property has been set to 1 second. So an idle NServiceBus role will slow down incrementally by 50 milliseconds until it polls only once every second. For our example of 2 roles with 4 threads, in idle state, this would now result in a cost of $0.0288 per hour.

Not so scary anymore 🙂

Please play with these values a bit, and let me know what you feel is the best balance between cost and performance…

Until next time…

Understanding the transactional behavior of an Azure Message Queue

As both Udi and Ayende mention, a remote queueing system (like azure queue storage) isn’t really comparable to a local queueing system (such as msmq).

One of the main differences is how they behave transactionally. Well in fact the remote queueing systems aren’t transactional at all, but support a timeout mechanism  to simulate transactions (often called queue-peek-lock mechanism).

The queue-peek-lock mechanism works in the following way:

  • Once a message is picked up by a process, the message will stay in the queue but will become invisible to other processes.
  • When the process is finished handling the message, it has to explicitly remove the message from the queue.
  • If the process fails to do so in a certain timeframe, because it crashed or isn’t done yet, the message will reappear on the queue.
  • And the message will be picked up by another process, executing the same logic again.

But hold on a minute! In your previous article you’re setting IsTransactional to true and now your telling me there are no real transactions, what’s up with that? Well, indeed, setting transactional to true means that we will play nicely along in this queue-peek-lock mechanism, setting it to false means we will immediately delete the message after receiving.

To avoid frequent occurrence of multi-processing you must ensure that your message handler completes before the timeout expires. By default the timeout is set to 30 seconds, but you can control it yourself using the MessageInvisibleTime property on the AzureQueueConfig configuration section (expressed in milliseconds).

But no matter what value you assign, multi-processing is always a potential problem on remote queueing systems, so make sure that your message handlers have been implemented in an idempotent way. In the future I hope to figure out a generic solution to this problem and add it to NServiceBus, but today I have no clue how to do it. I know Ayende divised the following solution using local transaction, but as azure table storage doesn’t support this kind of transactions it’s not really suitable in our case.

If you have any suggestion in this matter, I would, ofcourse, be happy to hear them…


Azure Queue Storage support for NServiceBus

NServiceBus

Some of you may know this already, but during the summer months in 2010 I’ve joined the NServiceBus team in order to add azure support to this open source communication framework.

Personally I feel NServiceBus is a great addition to any azure project, mostly because it has the ability to really simplify the way the different components of your azure solution communicate with one another. Especially the common communication patterns between web and worker roles, through azure queue storage or appfabric queues, become a breeze to implement, as you will see in the example a bit further on in this post.

Furthermore the implementation of the Saga pattern in NServiceBus can help you deal with the lack of distributed transactions, as those are not supported on azure (you knew this right?).

And to top things off, the framework has a lot of additional capabilities to help you implement your solution in a fast and simple way.

But all of these goodies are for later posts, lets start with the basics…

Full Duplex

The first example I would like to show is full duplex communication, aka asynchronous request reply, between a web and a worker role. I’ve choosen this communication pattern to show the azure queue storage support as it does not require any other feature of NServiceBus besides the message queue implementation. In future posts I will elaborate further on more of the NServiceBus features that are supported on azure. I do however NOT recommend this pattern to be used a lot in an azure solution, as it limits the scalability of your webroles. Because the return message is returned to the web role where the request originated and due to load balancing you may get some unexpected behaviour in your application when you scale out. But still it suits my ‘example’ purpose so let’s continue…

Using NServiceBus on Azure

Allright, let’s get started using NServiceBus in our example solution. (for which you can find the sources in the samples directory of the NServiceBus trunk, the sample is called AzureFullDuplex).

First thing to do, is make sure to that NServiceBus knows how to configure the environment and how you want your role to behave. This can be done very easily through the usage of profiles, all you need to do is implement a couple of marker interfaces on a class specifying how you want your role to behave. In our case we want a worker to answer the requests and a webrole to receive the results. In this case the worker just needs to listen for requests and answer them, doing this is as simple as

public class EndpointConfiguration : IConfigureThisEndpoint, AsA_Listener { }

I told you it was easy… .

Now we also need to start the NServiceBus runtime, in order to do so, I created a RoleEntryPoint base class in the NServiceBus.Hosting.Azure namespace for you to inherit from and it all happens magically.

public class Host : RoleEntryPoint{}

As mentioned, we support more than than 1 queueing mechanism azure storage queues and appfabric queues. This can be controlled by using profiles, in this case I’ve set the profile to

NServiceBus.Production NServiceBus.WithAzureStorageQueues

Allright, good to go…

Now, lets define the full duplex communication at both ends. The worker needs to contain a class that handles a request message (SubmitOrderRequest), the webrole needs a class that handles the response (SubmitOrderResponse) as well as you will see later. This class needs to implement the IHandleMessages<T> interface to tell NServiceBus about it’s capabilities.

Here is the code for the worker role’s capabilities. Note that NServiceBus has an ioc container internally, so if you need a reference to some implementation, you can just accept it in the constructor of your class. In this case we need a reference to the IBus implementation to reply on the incoming request. I guess the code is pretty self explanatory besides that…

public class SubmitOrderMessageHandler : IHandleMessages<SubmitOrderRequest>
{
    private readonly IBus bus;

    public SubmitOrderMessageHandler(IBus bus)
    {
        this.bus = bus;
    }

    public void Handle(SubmitOrderRequest message)
    {
        var order = new Order
                        {
                            Id = message.Id,
                            Quantity = message.Quantity,
                            Status = OrderStatus.Pending
                        };

        // in real life you would persist here of course

        var response = bus.CreateInstance(x=>x.Order = order);
        bus.Reply(response);
    }
}

Now at the web role side, we need to add two responsibilities which I have both implement in a webpage’s code behind (evil me). First of all, whenever the user submits the form we need to send the request to the worker. Secondly we need to accept the incoming response and update our local state with it. Do note however, that it is ‘Asynchronous’ request reply, this does mean that the response will come in any time AFTER the page has been rendered back to the user. A refresh mechanism will be required to make sure the user gets to see the results of his actions.

The code for this page looks something like this. Note asp.net has no ioc container, so I choose to work with static references to keep the sample simple (evil me again).

public partial class _Default : System.Web.UI.Page, IHandleMessages<SubmitOrderResponse>
{
    protected void Page_PreRender(object sender, EventArgs e)
    {
        lock (WebRole.Orders)
            OrderList.DataSource = new List<Order>(WebRole.Orders);

        OrderList.DataBind();
    }

    protected void btnSubmit_Click(object sender, EventArgs e)
    {
        WebRole.Bus
            .Send(new SubmitOrderRequest
            {
                Id = Guid.NewGuid(),
                Quantity = Convert.ToInt32(txtQuatity.Text)
            });
    }

    public void Handle(SubmitOrderResponse message)
    {
        lock (WebRole.Orders)
            WebRole.Orders.Add(message.Order);
    }
}

That’s all there is to it, from a coding point of view then, of course we still need to configure some values so that NServiceBus knows where to store it’s messages and how to poll for them (more on this in a later post).

Configuring NServiceBus

Configuring NServiceBus can be done in 3 ways when using it on azure. First of all you can configure it in the application configuration file. When hosting on azure this only makes sense for those properties that either do not change or for providing default values that can be overriden later in the service configuration file. When using azure queues from an on-premise app, this remains the default way of configuring the framework though.

You need a minimum of 3 config sections to get up and running

  • AzureQueueConfig contains the azure specific configuration settings such as the connectionstring.
  • MsmqTransportConfig although the name is a bit weird, due to backward compatibility, this section no longer has anything to do with msmq. It allows you to configure the behavior of the transport mechanism that uses a queue (in our case an azure queue instead of an msmq queue).
  • UnicastBusConfig contains configuration settings related to routing of messages, it allows you to specify where specific message types should be sent to.

An example would look like

<configSections>
  <section name="MsmqTransportConfig" type="NServiceBus.Config.MsmqTransportConfig, NServiceBus.Core"/>
  <section name="UnicastBusConfig" type="NServiceBus.Config.UnicastBusConfig, NServiceBus.Core"/>
  <section name="AzureQueueConfig" type="NServiceBus.Config.AzureQueueConfig, NServiceBus.Azure"/>
</configSections>

<MsmqTransportConfig NumberOfWorkerThreads="1" MaxRetries="5"/>

<AzureQueueConfig ConnectionString="UseDevelopmentStorage=true" />

<UnicastBusConfig LocalAddress="Orderwebsiteinputqueue">
  <MessageEndpointMappings>
   <add Messages="MyMessages" Endpoint="orderserviceinputqueue"/>
  </MessageEndpointMappings>
</UnicastBusConfig>

A second configuration option, available in windows azure only, is to configure NServiceBus from the service configuration file. Today it is not possible to configure everything in here as the service configuration file is far less advanced than a web.config file. But the majority of the settings can be configured though. My personal opinion is that you should use both together, and use the service configuration file to ‘override’ the defaults from the application configuration file.

In order to do so, you have to concatenate the name of the configuration setting with a dot and the property name. For example to override the connectionstring of the azure queue configuration settings you should specify the following:

<Setting name="AzureQueueConfig.ConnectionString" value="UseDevelopmentStorage=true" />

Finally you can also configure in code using a Fluent interface. I’ve set up the web client this way. I feel it’s quite valuable for you to know how to set up NServiceBus manually on top of azure. So here goes.

Bus = Configure.WithWeb()
            .DefaultBuilder() // sets up the ioc container
            .Log4Net(new AzureAppender()) // logging
            .AzureConfigurationSource() // configures service configuration file support
            .AzureMessageQueue() // use azure storage queues
                .JsonSerializer()
            .UnicastBus()
                .LoadMessageHandlers() // automatically register known message handlers
                .IsTransactional(true)
            .CreateBus()
            .Start();

Note: When hosting the web role in full IIS mode (which is the default) you can not access the HttpContext or RoleEnvironment from Application_Start. I’m using this internally, so you will have to postpone the configuration of the bus to Begin_Request but you need to ensure that it only gets executed on the first request. This can be easily achieved using the Lazy<T> construct in .

Now you should have enough information to get NServiceBus up and running in your azure solution. Please have a look at the sample, play with it and let me know about any issue you may run into.

Happy coding and stay tuned for more posts on this topic…

Logging with Log4Net on the Azure platform

Log4Net on Azure

I have always been a big fan of the Log4Net framework for all of my logging needs in applications and I would like to use this framework in my azure projects as well. And to be honest it isn’t even that difficult: just configure it to the TraceAppender as everything written to the tracelog will be transferred by the diagnostics manager to azure table storage. 

But there are two problems, or inconveniences, with this approach. First of all, you have to both configure log4net and the diagnostics monitor. This is a pretty repetitive task that you don’t want to perform over and over again for every application you create.  Secondly log4net is configured using the application’s configuration file, which is rather inconvenient on azure as it isn’t editable.

AzureAppender

To solve both of these problems we can instead create a custom appender that both sets up the diagnostics monitor as well as read the log4net configuration from the service configuration file instead of the application configuration file.

In order to do so, you have to inherit a class from AppenderSkeleton.

public sealed class AzureAppender : AppenderSkeleton

First thing that needs to be done is ensure that the configuration values are read from the service configuration file, if they are present. This is a bit clumbsy on azure as you cannot check for the presence of a configuration key, all you can do is act on the exception thrown when the key is not present. Make sure to set all values before the ActivateOptions method is called.

The following example shows you how to read the error level from config and apply it to the log4net environment.

private static string GetLevel()        
{            
       try            
       {                
           return RoleEnvironment.GetConfigurationSettingValue(LevelKey);            
       }           
       catch (Exception)            
       {                
            return "Error";            
       }        
}

 private void ConfigureThreshold()
 {
     var rootRepository = (Hierarchy) log4net.LogManager.GetRepository();
     Threshold = rootRepository.LevelMap[GetLevel()];
 }

The appender for this article supports the following configuration settings:

  • Diagnostics.ConnectionString Sets the connection string to be used when transferring the log entries to table storage
  • Diagnostics.Level Sets the threshold that log4net will use to filter the logs to output.
  • Diagnostics.Layout Defines the layout and content that log4net will use to create the log entries
  • Diagnostics.ScheduledTransferPeriod Specifies the interval, in minutes, that will be used by the diagnostics manager to transfer logs to azure table storage
  • Diagnostics.EventLogs Configures which of the event log sections will be transferred from the azure instance to azure table storage

When the options have been set and activated, the log4net environment has been completely configured to make proper use of our custom appender and we can start azure diagnostics monitor. Note that the diagnostics monitor also has a threshold that allows you to filter the logs written to storage. But as log4net is already filtering, we don’t need to do it here anymore so we set the filter to Verbose.

 private void ConfigureAzureDiagnostics()
{
    var traceListener = new DiagnosticMonitorTraceListener();
    Trace.Listeners.Add(traceListener);

    var dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();

    //set threshold to verbose, what gets logged is controled by the log4net level
    dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;

    ScheduleTransfer(dmc);

    ConfigureWindowsEventLogsToBeTransferred(dmc);

    DiagnosticMonitor.Start(ConnectionStringKey, dmc);
}

private void ScheduleTransfer(DiagnosticMonitorConfiguration dmc)
{
    var transferPeriod = TimeSpan.FromMinutes(ScheduledTransferPeriod);
    dmc.Logs.ScheduledTransferPeriod = transferPeriod;
    dmc.WindowsEventLog.ScheduledTransferPeriod = transferPeriod;
}

private static void ConfigureWindowsEventLogsToBeTransferred(DiagnosticMonitorConfiguration dmc)
{
    var eventLogs = GetEventLogs().Split(';');
    foreach (var log in eventLogs)
    {
         dmc.WindowsEventLog.DataSources.Add(log);
    }
}

That’s all there is to it basically, the only thing we need to do now is to apply the appender to the environment. This is done by creating an instance of the appender, configure it either in code or using the settings in the service configuration file, and finally configure the log4net environment.

var appender = new AzureAppender();
appender.ActivateOptions();
BasicConfigurator.Configure(appender);

You can find the source for this appender in the NServiceBus project and feel free to use it in your projects. I also want to give special credits to Andreas Ohlund for creating the first version of this appender.