Improving throughput with NServiceBus on Windows Azure

One of the things that has always bothered me personally on the ‘NServiceBus – Azure queue storage’ relationship is throughput, the amount of messages that I could transfer from one role to the other per second was rather limited.

This is mainly due to the fact that windows azure storage throttles you at the http level, every queue only accepts 500 http requests per second and will queue up the remaining requests. Given that you need 3 requests per message, you can see that throughput is quite limited, you can transfer less than a hundred messages per second. (Sending role performing 1 post request, receiving role performing 1 get and 1 delete request)

One of the first things that you can do to increase throughput is using the SendMessages() operation on the unicast bus.This operation will group all messages passed into it into 1 single message and send it across the wire. Mind that queue storage also limits message size to 8KB, so in effect you can achieve a maximum improvement of factor 10, given that you have reasonable small messages and use binary formatting.

Secondly I’ve added support to the queue for reading in batches, using the GetMessages operation on the cloud queue client. By default the queue reads 10 messages at a time, but you can use a new configuration setting called BatchSize to control the amount of messages to be read. Mind that the BatchSize setting also influences the MessageInvisibleTime, as I multiply this number by the batchsize to define how long the messages have to stay invisible as overall process time may now take longer.

In the future I may consider even more improvements to increase throughput of queue storage. Like for example using multiple queues at a time to overcome the 500 requests per second limit. But as Rinat Abdullin already pointed out to me on twitter this might have grave consequences on both overall latency and costs. So before I continue with this improvement I have a question for you, do you think this additional latency and costs are warranted?

But even then, there is another throttle in place at the storage account level, which limites all storage operation requests to 5000 requests per second (this includes table storage and blob storage requests), in order to work around this limit you can specify a separate connection string for every destination queue using the following format “queuename@connectionstring”.

Advertisements

3 Responses to Improving throughput with NServiceBus on Windows Azure

  1. Pete Hazelberg says:

    A little off topic, but when moving a high volume of large messages (where the Queues get paired with Table Storage), it might make sense to have an option to use SQL Azure to hold the larger messages instead of Azure Table Storage. That probably sounds like a strange option, but the reason is that SQL Azure does not charge per transaction so it can be a more cost effective storage mechanism at high transaction rates.

    With roughly 2.5M seconds/month, if you even averaged 100 large messages/sec with only 2 transactions per large message, that would be 500M table storage transactions/month. If my math is holding up, at $.01 per 10K transactions, the bill would be $500 (plus probably an insignificant $.15 per gigabyte stored). With those numbers of transactions, SQL Azure’s $10 per GB per month with unlimited transactions may be more cost effective in many cases.

    I haven’t gotten a chance to read many details on the new Azure AppFabric Service Bus Queues coming in v2, but if they don’t have a per-transaction charge, they may be more cost effective than Azure Storage Queues for high volume as well?

  2. Hi Pete,

    SQL Azure is indeed a good option, and I’m considering for NSB’s DataBus as well. But it is one that requires some thinking though as it throttles you fairly aggressively: there are only 30 queries per second allowed per connection if I’m not mistaken.

    Regarding these new queues there is no cost information available as of date, but I do assume that they will follow the same billing model as other servicebus constructs, so pay per connection instead of transaction. But as the queue size will be rather large (GB’s), I assume you will have to pay for the durable storage as well. (If I’m not mistaken this is a SQL Azure database as wel)…

    Kind regards,
    Yves

  3. Pingback: NServiceBus | Lplalonde's Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: