Azure Service Bus has long become a staple for distributed systems running on Microsoft Azure cloud. It is a high-throughput, reliable and easy to use messaging service that can be a backbone for your scalable solutions.
In this tutorial, we'll explore Service Bus integration with Azure Event Grid, and the advantages and scenarios it brings to the table.
Azure Event Grid integration
Integration with Event Grid is a fairly new functionality of the Service Bus, as this feature was launched in the autumn of 2018.
For the time being, it is only available for the Premium tier subscription for Service Bus - however if you are running any kind of reliable real-life production workflow, you probably already are on the premium tier, just because it is so much more reliable than Basic/Standard levels.
Event Grid integration adds a whole new dimension to architecting solutions against Service Bus, as it allows you to create message listeners or receivers that do not need to maintain a constant connection to the Service Bus.
Currently, Service Bus can raise an Event Grid event in two scenarios:
- active message available on a queue / topic subscription, without a message receiver connected (Microsoft.ServiceBus.ActiveMessagesAvailableWithNoListeners event type)
- dead letter messages available (Microsoft.ServiceBus.DeadletterMessagesAvailable event type)
More event types are expected to be added in the future.
How does it differ from traditional use cases?
In a typical Service Bus application integration pattern, for both queues (one-to-one messaging) and topic/subscriptions (one-to-many messaging), receivers maintain an open TCP connection with the Service Bus and listen to incoming messages using the AMQP protocol.
On rare occasions, if full bi-directional TCP communication cannot be established, the client may also continuously poll (long polling) the Service Bus for active messages.
Despite the fact that it is possible to have the receiver polling for messages, the best mental model for Service Bus usage though, is to treat it as a de facto "push model" that guarantees fast message delivery over an existing TCP connection.
The listener may connect early and be ready for any message that the producer would send, or it may choose to connect later, and simply pick up the active (pending) messages then - unprocessed messages are not lost, unless they expire.
This simple sequence flow is shown in Figure 1.
Figure 1: Typical Service Bus Architecture
Such an approach is suitable for most distributed system architectures, but the problem with it is that the listener, should it disconnect from the Service Bus, doesn't really know when the new messages show up. So, in order to be able to consume messages quickly, it normally ends up having to reconnect to the Service Bus straight away.
Azure Event Grid integration opens up a new possible integration path into Service Bus. Instead of the usual "push model", it flips the script and allows us to program against the Service Bus using a "pull model".
When using the Azure Event Grid integration, the receiver, on both queue and topic/subscription side of things, doesn't need to maintain an active TCP connection to the Service Bus anymore. Instead, it exposes a hook for the Azure Event Grid - this might be either HTTP webhook or an Event Hub hook.
For simplicity, we'll only focus on webhooks in this article.
The webhook is invoked by Azure Event Grid as soon as there is a message-related activity detected by the Service Bus. This gives the receiver a chance to connect to the Service Bus and handle the message.
We already called this approach a "pull model" but perhaps a more fitting name would be a "reactive system". The practical consequence is that we are able to handle a message on Service Bus, without having to maintain a constant connection to the Service Bus.
Just like with any software feature or functionality, you probably will come up with some of your own use cases where this behavior would come in handy - after all, it all depends on the context.
However, there are a few benefits of Azure Event Grid integration into Service Bus that immediately jump to mind, so let's briefly look at those.
Benefits of Azure Event Grid Integration into Service Bus
#Benefit 1: On demand message receivers
As already explained, the fact that you do not have to maintain permanent connection to Service Bus from your listener worker, can be very valuable.
It actually can be helpful in two ways.
First of all, you could apply certain cost or infrastructure saving measures by shutting down the listener processes when they are not in use, and only bringing them back online once Event Grid notifies you of new pending messages.
This is shown in a sequence diagram in Figure 2.
Figure 2: Service Bus Architecture with Event Grid
You can immediately see the difference compared to our previous, more traditional diagram (Figure 1), where a listener either had to be online all along, or it had to guess when a connection to Service Bus might be necessary.
In this case however, as soon as a new message that has no active listener shows up on a Service Bus queue or a topic subscription, Service Bus raises an Event Grid event about that.
Event Grid would then call the web hook with that information, into something that we call a "Message Watchdog" (see Figure 2). It will be a custom component that we explicitly introduce to listen to Azure Event Grid events and perform certain actions based on that.
“Message Watchdog” is of course a generic name and depending on your system architecture, the watchdog process could take different shapes and forms – it is up to you to structure your system landscape in a way that suits your business and technical needs.
For example, the watchdog functionality could be fulfilled by a monitoring system that you always have online. It could also be fulfilled by another part of the service landscape, which contrary to the message listener isn't getting shut down, such as an API server. It could also be completely serverless, i.e. an Azure Function.
In our simple example, the watchdog is then responsible for bringing the message consumer (listener) into the picture, by notifying it that it is time to connect to Service Bus to pick up the message, provided the consumer is already alive. There’s also the possibility where the watchdog has to orchestrate the provisioning of the message consumer if it's actually offline or deprovisioned.
The second benefit of this type of Event Grid powered integration into Service Bus, is that it allows you to bypass the connection limits of Service Bus namespaces.
At the moment, in AMQP mode, the limit is 5000 concurrent connections per Service Bus namespace. This might seem like a lot, but that all depends on the system architecture.
It could be - and I have seen this in some enterprise-integration scenarios - that the number of connected listeners can be really high. This could be caused by large scale out situations, by having many specialized message types, that can only be handled by a specific type of listener. This could also be - and this is something to definitely avoid - found in applications created for internal enterprise use, allowing client applications to connect to the Service Bus namespace directly.
#Benefit 2: Monitoring message receivers
Another useful use case for the Event Grid integration, is to combine the traditional "push model" of Service Bus usage, with Event Grid web hook notification used as a reliable self-heal monitoring feature.
In such an architecture, we'd still use Service Bus the same way as before (as shown on Figure 1), but we'd leverage Event Grid to notify us that there are active messages without a listener.
This could happen when, for example, our message receiver crashes or loses network connectivity. In such situations, Event Grid would notify our Message Watchdog, which could then spin up a new instance of the receiver, thus equipping us with a nice self-healing system.
This process is shown in Figure 3.
Figure 3: Service Bus Architecture with Event Grid
#Benefit 3: Dead letter queue monitoring
Finally, the Event Grid integration provides an excellent way for keeping tab on the dead letter queue - be it for a regular Service Bus queue, or for a topic subscription (since it will have its own dedicated dead letter queue).
Since dead letter queue is well, a queue, you could of course connect to it and monitor it for new messages all the time - and be able to notify your monitoring or support teams when a new dead letter message shows up.
This is somewhat wasteful though, since you need to be connected all the time in order to be able to timely react to the (hopefully) occasional dead letter message. The alternative is to check the dead letter queue at regular intervals, continuously reconnecting and disconnecting to it, but of course that means that your reaction time might be slowed down.
With Event Grid, things get much cleaner.
Just like shown in the earlier diagrams, Service Bus is able to raise an Event Grid event when there is a new message in the new dead letter queue, which normally means something went really wrong in the message consumption process. At that point, you are free to react to that message however you wish - without having to continuously observe the dead letter queue.
Azure Event Grid Integration - Trying it out
Now that we have a long and winding theoretical road behind us, it's time to try this feature out.
We'll assume here that you are already familiar with Azure Service Bus and can provision a new Service Bus instance for your use.
In case you have troubles, or need a refresher, there is an excellent "Get started with Service Bus topics" tutorial. It specifically deals with topics, because that is what we are going to be using here, but it's very similar for queues.
Once your Azure Service Bus namespace is created, and your queue is ready (remember, you need a Premium tier Service Bus!), let's set up the Azure Event Grid integration. This is done by going to the Events blade of your Service Bus namespace as shown in Figure 4.
Figure 4: Event Grid Integration Setup
Note that the configuration of Event Grid happens against the namespace, even if you'd like to listen to events from a specific queue or a specific subscription only. In those cases, you'd apply special filtering which we'll be looking at soon.
For now, click +Event Subscription (see Figure 4) to create a new Event Grid integration. This should take you to the next screen as shown in Figure 5.
Figure 5: Create New Event Grid Integration
In Figure 6, we choose the event type we are interested in - in our example only Microsoft.ServiceBus.ActiveMessagesAvailableWithNoListeners and we choose where the event should be directed at.
In this particular example, it's a webhook with a test service that I have created to receive the Event Grid events (so something corresponding to the "Messaging Watchdog" from our sequence diagrams).
At this point, the event configuration would apply to the entire Service Bus namespace – i.e. all queues and all topics/subscriptions.
Should you wish to constraint the events to a specific set of Service Bus entities, you can switch to the Filter tab as shown in Figure 6.
Figure 6: Applying Filters to the Subject of Each Event
While not tremendously intuitive, this tab allows you to pick the entity you are interested in. This is done using the Subject Ends With configuration field, where you can put in the name of your queue or the name of your topic with subscription (that one in the format <topic name>/subscriptions/<subscription name>).
At this point, we can save the whole Event Grid configuration and try this out. Overall, my demo setup uses the following names:
Given such naming style, Event Grid, in case there is an active message that is not being listened to by any receiver, will raise the following ActiveMessagesAvailableWithNoListeners event, which we'll be able to consume in our watchdog process.
[
{
"data": {
"entityType": "subscriber",
"namespaceName": "strathweb-test",
"queueName": null,
"requestUri": "https://strathweb-test.servicebus.windows.net/event-grid-demo/subscriptions/strathweb-subscription/messages/head",
"subscriptionName": "strathweb-subscription",
"topicName": "event-grid-demo"
},
"dataVersion": "1",
"eventTime": "2019-03-25T20:33:45.5378923Z",
"eventType": "Microsoft.ServiceBus.ActiveMessagesAvailableWithNoListeners",
"id": "f8b78b1b-593a-41ca-95ee-80f59d5ee3f9",
"metadataVersion": "1",
"subject": "topics/event-grid-demo/subscriptions/strathweb-subscription",
"topic": "/subscriptions/59027101-5bc0-4c11-aa7a-0f66f7054bd3/resourcegroups/strathweb-demos/providers/Microsoft.ServiceBus/namespaces/strathweb-test"
}
]
This means that you can now create your watchdog process as an API endpoint. It doesn't really matter what endpoint it is - an ASP.NET Core app on Azure App Service, a Logic App, an Azure Function and so on. You can then handle this incoming JSON, and make logical, educated system decisions like the ones we discussed before.
Please note - and this is out of scope for this article - in order to consume Event Grid events in an API application, you must first go through a simple validation process (basically consume another JSON payload once before). This is described in details on docs.microsoft.com.
Summary
I very much encourage you to try out Service Bus with Event Grid integration.
It allows us to go beyond the traditional Service Bus system architectures and build truly reactive, event driven systems, where message receivers can come to existence, and go away on demand, as new messages flow through your system landscape.
This model wonderfully supplements the established messaging patterns with Service Bus, enhancing the toolkit of every system architect relying on Azure for messaging.
This article was technically reviewed by Tim Sommer.
This article has been editorially reviewed by Suprotim Agarwal.
C# and .NET have been around for a very long time, but their constant growth means there’s always more to learn.
We at DotNetCurry are very excited to announce The Absolutely Awesome Book on C# and .NET. This is a 500 pages concise technical eBook available in PDF, ePub (iPad), and Mobi (Kindle).
Organized around concepts, this Book aims to provide a concise, yet solid foundation in C# and .NET, covering C# 6.0, C# 7.0 and .NET Core, with chapters on the latest .NET Core 3.0, .NET Standard and C# 8.0 (final release) too. Use these concepts to deepen your existing knowledge of C# and .NET, to have a solid grasp of the latest in C# and .NET OR to crack your next .NET Interview.
Click here to Explore the Table of Contents or Download Sample Chapters!
Was this article worth reading? Share it with fellow developers too. Thanks!
Filip is a founder and maintainer of several popular open source projects, .NET blogger, author and a Microsoft MVP. He specializes in the Roslyn compiler, cross platform .NET development, ASP.NET Core and is experienced in delivering robust web solutions across the globe.Filip is based in Zürich where he works in a medical industry for Sonova AG. You can follow him on Twitter < a href = 'http://twitter.com/filip_woj' target='_blank'> @filip_woj .