Azure Queue Storage support for NServiceBus

NServiceBus

Some of you may know this already, but during the summer months in 2010 I’ve joined the NServiceBus team in order to add azure support to this open source communication framework.

Personally I feel NServiceBus is a great addition to any azure project, mostly because it has the ability to really simplify the way the different components of your azure solution communicate with one another. Especially the common communication patterns between web and worker roles, through azure queue storage or appfabric queues, become a breeze to implement, as you will see in the example a bit further on in this post.

Furthermore the implementation of the Saga pattern in NServiceBus can help you deal with the lack of distributed transactions, as those are not supported on azure (you knew this right?).

And to top things off, the framework has a lot of additional capabilities to help you implement your solution in a fast and simple way.

But all of these goodies are for later posts, lets start with the basics…

Full Duplex

The first example I would like to show is full duplex communication, aka asynchronous request reply, between a web and a worker role. I’ve choosen this communication pattern to show the azure queue storage support as it does not require any other feature of NServiceBus besides the message queue implementation. In future posts I will elaborate further on more of the NServiceBus features that are supported on azure. I do however NOT recommend this pattern to be used a lot in an azure solution, as it limits the scalability of your webroles. Because the return message is returned to the web role where the request originated and due to load balancing you may get some unexpected behaviour in your application when you scale out. But still it suits my ‘example’ purpose so let’s continue…

Using NServiceBus on Azure

Allright, let’s get started using NServiceBus in our example solution. (for which you can find the sources in the samples directory of the NServiceBus trunk, the sample is called AzureFullDuplex).

First thing to do, is make sure to that NServiceBus knows how to configure the environment and how you want your role to behave. This can be done very easily through the usage of profiles, all you need to do is implement a couple of marker interfaces on a class specifying how you want your role to behave. In our case we want a worker to answer the requests and a webrole to receive the results. In this case the worker just needs to listen for requests and answer them, doing this is as simple as

public class EndpointConfiguration : IConfigureThisEndpoint, AsA_Listener { }

I told you it was easy… .

Now we also need to start the NServiceBus runtime, in order to do so, I created a RoleEntryPoint base class in the NServiceBus.Hosting.Azure namespace for you to inherit from and it all happens magically.

public class Host : RoleEntryPoint{}

As mentioned, we support more than than 1 queueing mechanism azure storage queues and appfabric queues. This can be controlled by using profiles, in this case I’ve set the profile to

NServiceBus.Production NServiceBus.WithAzureStorageQueues

Allright, good to go…

Now, lets define the full duplex communication at both ends. The worker needs to contain a class that handles a request message (SubmitOrderRequest), the webrole needs a class that handles the response (SubmitOrderResponse) as well as you will see later. This class needs to implement the IHandleMessages<T> interface to tell NServiceBus about it’s capabilities.

Here is the code for the worker role’s capabilities. Note that NServiceBus has an ioc container internally, so if you need a reference to some implementation, you can just accept it in the constructor of your class. In this case we need a reference to the IBus implementation to reply on the incoming request. I guess the code is pretty self explanatory besides that…

public class SubmitOrderMessageHandler : IHandleMessages<SubmitOrderRequest>
{
    private readonly IBus bus;

    public SubmitOrderMessageHandler(IBus bus)
    {
        this.bus = bus;
    }

    public void Handle(SubmitOrderRequest message)
    {
        var order = new Order
                        {
                            Id = message.Id,
                            Quantity = message.Quantity,
                            Status = OrderStatus.Pending
                        };

        // in real life you would persist here of course

        var response = bus.CreateInstance(x=>x.Order = order);
        bus.Reply(response);
    }
}

Now at the web role side, we need to add two responsibilities which I have both implement in a webpage’s code behind (evil me). First of all, whenever the user submits the form we need to send the request to the worker. Secondly we need to accept the incoming response and update our local state with it. Do note however, that it is ‘Asynchronous’ request reply, this does mean that the response will come in any time AFTER the page has been rendered back to the user. A refresh mechanism will be required to make sure the user gets to see the results of his actions.

The code for this page looks something like this. Note asp.net has no ioc container, so I choose to work with static references to keep the sample simple (evil me again).

public partial class _Default : System.Web.UI.Page, IHandleMessages<SubmitOrderResponse>
{
    protected void Page_PreRender(object sender, EventArgs e)
    {
        lock (WebRole.Orders)
            OrderList.DataSource = new List<Order>(WebRole.Orders);

        OrderList.DataBind();
    }

    protected void btnSubmit_Click(object sender, EventArgs e)
    {
        WebRole.Bus
            .Send(new SubmitOrderRequest
            {
                Id = Guid.NewGuid(),
                Quantity = Convert.ToInt32(txtQuatity.Text)
            });
    }

    public void Handle(SubmitOrderResponse message)
    {
        lock (WebRole.Orders)
            WebRole.Orders.Add(message.Order);
    }
}

That’s all there is to it, from a coding point of view then, of course we still need to configure some values so that NServiceBus knows where to store it’s messages and how to poll for them (more on this in a later post).

Configuring NServiceBus

Configuring NServiceBus can be done in 3 ways when using it on azure. First of all you can configure it in the application configuration file. When hosting on azure this only makes sense for those properties that either do not change or for providing default values that can be overriden later in the service configuration file. When using azure queues from an on-premise app, this remains the default way of configuring the framework though.

You need a minimum of 3 config sections to get up and running

  • AzureQueueConfig contains the azure specific configuration settings such as the connectionstring.
  • MsmqTransportConfig although the name is a bit weird, due to backward compatibility, this section no longer has anything to do with msmq. It allows you to configure the behavior of the transport mechanism that uses a queue (in our case an azure queue instead of an msmq queue).
  • UnicastBusConfig contains configuration settings related to routing of messages, it allows you to specify where specific message types should be sent to.

An example would look like

<configSections>
  <section name="MsmqTransportConfig" type="NServiceBus.Config.MsmqTransportConfig, NServiceBus.Core"/>
  <section name="UnicastBusConfig" type="NServiceBus.Config.UnicastBusConfig, NServiceBus.Core"/>
  <section name="AzureQueueConfig" type="NServiceBus.Config.AzureQueueConfig, NServiceBus.Azure"/>
</configSections>

<MsmqTransportConfig NumberOfWorkerThreads="1" MaxRetries="5"/>

<AzureQueueConfig ConnectionString="UseDevelopmentStorage=true" />

<UnicastBusConfig LocalAddress="Orderwebsiteinputqueue">
  <MessageEndpointMappings>
   <add Messages="MyMessages" Endpoint="orderserviceinputqueue"/>
  </MessageEndpointMappings>
</UnicastBusConfig>

A second configuration option, available in windows azure only, is to configure NServiceBus from the service configuration file. Today it is not possible to configure everything in here as the service configuration file is far less advanced than a web.config file. But the majority of the settings can be configured though. My personal opinion is that you should use both together, and use the service configuration file to ‘override’ the defaults from the application configuration file.

In order to do so, you have to concatenate the name of the configuration setting with a dot and the property name. For example to override the connectionstring of the azure queue configuration settings you should specify the following:

<Setting name="AzureQueueConfig.ConnectionString" value="UseDevelopmentStorage=true" />

Finally you can also configure in code using a Fluent interface. I’ve set up the web client this way. I feel it’s quite valuable for you to know how to set up NServiceBus manually on top of azure. So here goes.

Bus = Configure.WithWeb()
            .DefaultBuilder() // sets up the ioc container
            .Log4Net(new AzureAppender()) // logging
            .AzureConfigurationSource() // configures service configuration file support
            .AzureMessageQueue() // use azure storage queues
                .JsonSerializer()
            .UnicastBus()
                .LoadMessageHandlers() // automatically register known message handlers
                .IsTransactional(true)
            .CreateBus()
            .Start();

Note: When hosting the web role in full IIS mode (which is the default) you can not access the HttpContext or RoleEnvironment from Application_Start. I’m using this internally, so you will have to postpone the configuration of the bus to Begin_Request but you need to ensure that it only gets executed on the first request. This can be easily achieved using the Lazy<T> construct in .

Now you should have enough information to get NServiceBus up and running in your azure solution. Please have a look at the sample, play with it and let me know about any issue you may run into.

Happy coding and stay tuned for more posts on this topic…

Advertisements

16 Responses to Azure Queue Storage support for NServiceBus

  1. Pingback: NServiceBus Updates

  2. Pingback: Understanding the transactional behavior of an Azure Message Queue « Cloud Shaper

  3. ash says:

    I cannot run the samples. It fails to configure the orderserviceinputqueue.

  4. Hans says:

    Hi Goeleven,

    Can you provide me the steps to get the sample working on my machine?

    I downloaded the sample but what do still need to configure to get the sample running, when I hit f5 I get an error

    InvalidOperationException: No destination specified for message MyMessages.SubmitOrderRequest. Message cannot be sent. Check the UnicastBusConfig section in your config file and ensure that a MessageEndpointMapping exists for the message type.]

  5. Hans says:

    Hi Goeleven,

    Can you provide me the steps to get the sample working on my machine?

    I downloaded the sample but what do still need to configure to get the sample running, when I hit f5 I get an error

    InvalidOperationException: No destination specified for message MyMessages.SubmitOrderRequest. Message cannot be sent. Check the UnicastBusConfig section in your config file and ensure that a MessageEndpointMapping exists for the message type.]

    Any help would be appreciated!

    Hans

  6. Guys,

    You should be able to run the sample by simply hitting F5, no configuration required…

    I’ve just wiped everything from my computer, pulled a clean version from github and hit F5… worked like a charm…

    I’ll ask some other people to try the sample, see if we can reproduce this issue…

  7. flavius says:

    Hi Yves, I am having issues running the sample because the MyMessages folder is empty. I currently have revision 1416. Should I take the messages from somewhere else?

  8. Hi flavius, that is correct for your revision, but in the mean time the file has been added (and a lot of other stuff as well), Can you please update to a later revision?

  9. chutch says:

    Is it possible to use azure message queues, but from an on premise host (ie, a host that is running in the nservervicebus.host.exe process)? My problem is that my sender is in a webrole (works with nservicebus no problem) and my receiver is required to be on premise. I was going down a azure connect to wcf service path, but would rather use azure queues.

    • Hi Chad,

      That is definitly possible, in fact a lot of the samples are built that way: console apps that use azure storage queues from on-premise. There is no profile for this configuration in the host though, so you will have to do some custom initialization.

      Kind regards,
      Yves

      • chutch says:

        I see. So does that also mean that i will not be able to install it as a windows service (using NServiceBus.Host.exe /install) as well?

        Thanks for your assistance.
        Chad

      • Think you misunderstood, you can’t use the azure profiles in an on-prem host, but you can still use the on-prem profiles… but you will have to override msmq by the azure message queue by implementing IWantCustomInitialization.

        This also allows you to install the host as a windows service…

      • chutch says:

        So something like this should be enough then?

        internal class EndpointConfig : IConfigureThisEndpoint, AsA_Server, IWantCustomInitialization
        {
        #region IWantCustomInitialization Members

        public void Init()
        {
        Configure.With()
        .DefaultBuilder()
        .AzureMessageQueue()
        .JsonSerializer()
        .UnicastBus()
        .IsTransactional(true)
        .InMemorySubscriptionStorage();
        }

        #endregion
        }

      • Yes, think that should do…

        Guess you can limit it to AzureMessageQueue and JSonSerializer as the other setting are defaults imho

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: