Operational costs of an Azure Message Queue

A potential issue with azure message queues is the fact that there are hidden costs associated with them, depending on how often the system accesses the queues you pay more, no matter whether there are messages or not.

Lets have a look at what the operational cost model for NServiceBus would look like if we treated azure queues the same as we do msmq queues.

On azure you pay $0.01 per 10K storage transactions. Every message send is a transaction, every successful message read are 2 transactions (GET + DELETE) and also every poll without a resulting message is a transaction. Multiply this by the number of roles you have, and again by the number of threads in the role and again by the number of polls the cpu can launch in an infinite for-loop per second and you’ve got yourself quite a bill at the end of the month. For example, 2 roles with 4 threads each, in idle state, at a  rate of 100 polls per second would result in an additional $2.88 per hour for accessing your queues.

Scary isn’t it 🙂

As a modern day developer you have the additional responsibility to take these operational costs into account and balance them against other requirements such as performance. In order for you to achieve the correct balance, I’ve implemented a configurable back-off mechanism in the NServiceBus implementation.

The basic idea is, if the thread has just processed a message it’s very likely that there are more messages, so we check the queue again in order to maintain high throughput. But if there was no message, we will delay our next read a bit before checking again. If there is still no message, we delay the next read a little more, and so on and so on, until a certain threshold has been reached. This poll interval will be maintained until there is a new message on the queue. Once a message has been processed we start the entire sequence again from zero…

Configuring the increment of the poll interval can be done by setting the PollIntervalIncrement property (in milliseconds) on the AzureQueueConfig configuration section, by default the interval is increased by 50 milliseconds at a time. To define the maximum wait time when there are no messages, you can configure the MaximumWaitTimeWhenIdle property (also in milliseconds). By default this property has been set to 1 second. So an idle NServiceBus role will slow down incrementally by 50 milliseconds until it polls only once every second. For our example of 2 roles with 4 threads, in idle state, this would now result in a cost of $0.0288 per hour.

Not so scary anymore 🙂

Please play with these values a bit, and let me know what you feel is the best balance between cost and performance…

Until next time…

Advertisements

Understanding the transactional behavior of an Azure Message Queue

As both Udi and Ayende mention, a remote queueing system (like azure queue storage) isn’t really comparable to a local queueing system (such as msmq).

One of the main differences is how they behave transactionally. Well in fact the remote queueing systems aren’t transactional at all, but support a timeout mechanism  to simulate transactions (often called queue-peek-lock mechanism).

The queue-peek-lock mechanism works in the following way:

  • Once a message is picked up by a process, the message will stay in the queue but will become invisible to other processes.
  • When the process is finished handling the message, it has to explicitly remove the message from the queue.
  • If the process fails to do so in a certain timeframe, because it crashed or isn’t done yet, the message will reappear on the queue.
  • And the message will be picked up by another process, executing the same logic again.

But hold on a minute! In your previous article you’re setting IsTransactional to true and now your telling me there are no real transactions, what’s up with that? Well, indeed, setting transactional to true means that we will play nicely along in this queue-peek-lock mechanism, setting it to false means we will immediately delete the message after receiving.

To avoid frequent occurrence of multi-processing you must ensure that your message handler completes before the timeout expires. By default the timeout is set to 30 seconds, but you can control it yourself using the MessageInvisibleTime property on the AzureQueueConfig configuration section (expressed in milliseconds).

But no matter what value you assign, multi-processing is always a potential problem on remote queueing systems, so make sure that your message handlers have been implemented in an idempotent way. In the future I hope to figure out a generic solution to this problem and add it to NServiceBus, but today I have no clue how to do it. I know Ayende divised the following solution using local transaction, but as azure table storage doesn’t support this kind of transactions it’s not really suitable in our case.

If you have any suggestion in this matter, I would, ofcourse, be happy to hear them…


Azure Queue Storage support for NServiceBus

NServiceBus

Some of you may know this already, but during the summer months in 2010 I’ve joined the NServiceBus team in order to add azure support to this open source communication framework.

Personally I feel NServiceBus is a great addition to any azure project, mostly because it has the ability to really simplify the way the different components of your azure solution communicate with one another. Especially the common communication patterns between web and worker roles, through azure queue storage or appfabric queues, become a breeze to implement, as you will see in the example a bit further on in this post.

Furthermore the implementation of the Saga pattern in NServiceBus can help you deal with the lack of distributed transactions, as those are not supported on azure (you knew this right?).

And to top things off, the framework has a lot of additional capabilities to help you implement your solution in a fast and simple way.

But all of these goodies are for later posts, lets start with the basics…

Full Duplex

The first example I would like to show is full duplex communication, aka asynchronous request reply, between a web and a worker role. I’ve choosen this communication pattern to show the azure queue storage support as it does not require any other feature of NServiceBus besides the message queue implementation. In future posts I will elaborate further on more of the NServiceBus features that are supported on azure. I do however NOT recommend this pattern to be used a lot in an azure solution, as it limits the scalability of your webroles. Because the return message is returned to the web role where the request originated and due to load balancing you may get some unexpected behaviour in your application when you scale out. But still it suits my ‘example’ purpose so let’s continue…

Using NServiceBus on Azure

Allright, let’s get started using NServiceBus in our example solution. (for which you can find the sources in the samples directory of the NServiceBus trunk, the sample is called AzureFullDuplex).

First thing to do, is make sure to that NServiceBus knows how to configure the environment and how you want your role to behave. This can be done very easily through the usage of profiles, all you need to do is implement a couple of marker interfaces on a class specifying how you want your role to behave. In our case we want a worker to answer the requests and a webrole to receive the results. In this case the worker just needs to listen for requests and answer them, doing this is as simple as

public class EndpointConfiguration : IConfigureThisEndpoint, AsA_Listener { }

I told you it was easy… .

Now we also need to start the NServiceBus runtime, in order to do so, I created a RoleEntryPoint base class in the NServiceBus.Hosting.Azure namespace for you to inherit from and it all happens magically.

public class Host : RoleEntryPoint{}

As mentioned, we support more than than 1 queueing mechanism azure storage queues and appfabric queues. This can be controlled by using profiles, in this case I’ve set the profile to

NServiceBus.Production NServiceBus.WithAzureStorageQueues

Allright, good to go…

Now, lets define the full duplex communication at both ends. The worker needs to contain a class that handles a request message (SubmitOrderRequest), the webrole needs a class that handles the response (SubmitOrderResponse) as well as you will see later. This class needs to implement the IHandleMessages<T> interface to tell NServiceBus about it’s capabilities.

Here is the code for the worker role’s capabilities. Note that NServiceBus has an ioc container internally, so if you need a reference to some implementation, you can just accept it in the constructor of your class. In this case we need a reference to the IBus implementation to reply on the incoming request. I guess the code is pretty self explanatory besides that…

public class SubmitOrderMessageHandler : IHandleMessages<SubmitOrderRequest>
{
    private readonly IBus bus;

    public SubmitOrderMessageHandler(IBus bus)
    {
        this.bus = bus;
    }

    public void Handle(SubmitOrderRequest message)
    {
        var order = new Order
                        {
                            Id = message.Id,
                            Quantity = message.Quantity,
                            Status = OrderStatus.Pending
                        };

        // in real life you would persist here of course

        var response = bus.CreateInstance(x=>x.Order = order);
        bus.Reply(response);
    }
}

Now at the web role side, we need to add two responsibilities which I have both implement in a webpage’s code behind (evil me). First of all, whenever the user submits the form we need to send the request to the worker. Secondly we need to accept the incoming response and update our local state with it. Do note however, that it is ‘Asynchronous’ request reply, this does mean that the response will come in any time AFTER the page has been rendered back to the user. A refresh mechanism will be required to make sure the user gets to see the results of his actions.

The code for this page looks something like this. Note asp.net has no ioc container, so I choose to work with static references to keep the sample simple (evil me again).

public partial class _Default : System.Web.UI.Page, IHandleMessages<SubmitOrderResponse>
{
    protected void Page_PreRender(object sender, EventArgs e)
    {
        lock (WebRole.Orders)
            OrderList.DataSource = new List<Order>(WebRole.Orders);

        OrderList.DataBind();
    }

    protected void btnSubmit_Click(object sender, EventArgs e)
    {
        WebRole.Bus
            .Send(new SubmitOrderRequest
            {
                Id = Guid.NewGuid(),
                Quantity = Convert.ToInt32(txtQuatity.Text)
            });
    }

    public void Handle(SubmitOrderResponse message)
    {
        lock (WebRole.Orders)
            WebRole.Orders.Add(message.Order);
    }
}

That’s all there is to it, from a coding point of view then, of course we still need to configure some values so that NServiceBus knows where to store it’s messages and how to poll for them (more on this in a later post).

Configuring NServiceBus

Configuring NServiceBus can be done in 3 ways when using it on azure. First of all you can configure it in the application configuration file. When hosting on azure this only makes sense for those properties that either do not change or for providing default values that can be overriden later in the service configuration file. When using azure queues from an on-premise app, this remains the default way of configuring the framework though.

You need a minimum of 3 config sections to get up and running

  • AzureQueueConfig contains the azure specific configuration settings such as the connectionstring.
  • MsmqTransportConfig although the name is a bit weird, due to backward compatibility, this section no longer has anything to do with msmq. It allows you to configure the behavior of the transport mechanism that uses a queue (in our case an azure queue instead of an msmq queue).
  • UnicastBusConfig contains configuration settings related to routing of messages, it allows you to specify where specific message types should be sent to.

An example would look like

<configSections>
  <section name="MsmqTransportConfig" type="NServiceBus.Config.MsmqTransportConfig, NServiceBus.Core"/>
  <section name="UnicastBusConfig" type="NServiceBus.Config.UnicastBusConfig, NServiceBus.Core"/>
  <section name="AzureQueueConfig" type="NServiceBus.Config.AzureQueueConfig, NServiceBus.Azure"/>
</configSections>

<MsmqTransportConfig NumberOfWorkerThreads="1" MaxRetries="5"/>

<AzureQueueConfig ConnectionString="UseDevelopmentStorage=true" />

<UnicastBusConfig LocalAddress="Orderwebsiteinputqueue">
  <MessageEndpointMappings>
   <add Messages="MyMessages" Endpoint="orderserviceinputqueue"/>
  </MessageEndpointMappings>
</UnicastBusConfig>

A second configuration option, available in windows azure only, is to configure NServiceBus from the service configuration file. Today it is not possible to configure everything in here as the service configuration file is far less advanced than a web.config file. But the majority of the settings can be configured though. My personal opinion is that you should use both together, and use the service configuration file to ‘override’ the defaults from the application configuration file.

In order to do so, you have to concatenate the name of the configuration setting with a dot and the property name. For example to override the connectionstring of the azure queue configuration settings you should specify the following:

<Setting name="AzureQueueConfig.ConnectionString" value="UseDevelopmentStorage=true" />

Finally you can also configure in code using a Fluent interface. I’ve set up the web client this way. I feel it’s quite valuable for you to know how to set up NServiceBus manually on top of azure. So here goes.

Bus = Configure.WithWeb()
            .DefaultBuilder() // sets up the ioc container
            .Log4Net(new AzureAppender()) // logging
            .AzureConfigurationSource() // configures service configuration file support
            .AzureMessageQueue() // use azure storage queues
                .JsonSerializer()
            .UnicastBus()
                .LoadMessageHandlers() // automatically register known message handlers
                .IsTransactional(true)
            .CreateBus()
            .Start();

Note: When hosting the web role in full IIS mode (which is the default) you can not access the HttpContext or RoleEnvironment from Application_Start. I’m using this internally, so you will have to postpone the configuration of the bus to Begin_Request but you need to ensure that it only gets executed on the first request. This can be easily achieved using the Lazy<T> construct in .

Now you should have enough information to get NServiceBus up and running in your azure solution. Please have a look at the sample, play with it and let me know about any issue you may run into.

Happy coding and stay tuned for more posts on this topic…