CODEWITHSUNDEEP
azureazure-eventhub
Nikesh Devaki
Nikesh Devaki

Reputation: 2169

Available methods to monitor Azure Event hub parititons queue size

Kafka provides capability to monitor current offset and latest offset. Similarly does azure eventhub expose any api to continously monitor partition's current offset and latest available offset?

Upvotes: 0

Views: 2150

Answers (3)

Stefan Hudelmaier
Stefan Hudelmaier

Reputation: 151

I assume that your intent is to monitor the "queue size" of the Event Hub, aka the lag (the difference between the current offset and the latest offset in the context of a consumer group). There are different possibilities for doing this, depending on the SKU of your Event Hub.

Use Event Hub Premium and Dedicated Tiers

Event Hub offers a built-in metric for the lag in the Premium and Dedicated SKUs. The way this works is that you configure Diagnostic Settings for your Event Hub Namespace or cluster and send the Application Metrics Logs category to a Log Analytics Workspace. Here is the official documentation for this. I have written a how-to, including an example for monitoring this metric using an Azure Monitor alert, which you can find here.

Use an app from the Azure Marketplace

If you are on the Basic or Standard SKU and you do not want to implement and maintain this yourself (see below), you can use this offer from the Azure Marketplace: Lag Metrics for Event Hubs.

It automatically retrieves all Event Hubs of a namespace, creates lag metrics for all consumer groups and sends them to Application Insights / Log Analytics Workspace. An example for deploying this and for configuring Alert rules can be found on GitHub.

Disclaimer: I am the author of this app

Implement the metric yourself

  • Here is an example implementation in C#.
  • Here is an example implementation in Typescript.
  • There is also ablog post from Microsoft that also explains this, including an example in C#.

Upvotes: 4

Pankaj Rawat
Pankaj Rawat

Reputation: 4573

Extending above answer, you can see offset by 2 ways.

  1. Print offset in log file where you are listening EventHub

e.g using Azure function

    public static async Task Run([EventHubTrigger("EventHubname", ConsumerGroup = "ConsumerGroupname", Connection = "EventHubConnection")]EventData eventMessage,
        [Inject]IService service, [Inject]ILog log)
    {
    log.Info($"PartitionKey {eventMessage.PartitionKey}, Offset {eventMessage.Offset} and SequenceNumber {eventMessage.SequenceNumber}");
    }
  1. I am listening Eventhub by Azure Functions, You can see below location where Azure function maintain offset by partition.

From Portal

Option 3 (Latest)

Offset is not the correct way to measure the depth of Eventhub, Specially when you want to check how many messages need to process.

Now We are using Eventhub message SequenceNumber instead Offset. We have created TimerTrigger Azure Function. On every 5 minutes, we are getting LastEnqueuedSequenceNumber from Eventhub and SequenceNumber for each partition from blob storage (checkpoint location), then we storing difference in ApplicationInsight customMetrics.

Then ApplicationInsights help us to PIN Eventhub depth information in Azure dashboard and set up an alert.

Eventhub Depth in Azure Dashboard

Timer Trigger Code

I hope this will help!

Upvotes: 2

rickvdbosch
rickvdbosch

Reputation: 15551

Looking at Features and terminology in Azure Event Hubs - Event consumers - Stream offsets:

An offset is the position of an event within a partition. You can think of an offset as a client-side cursor. The offset is a byte numbering of the event. This offset enables an event consumer (reader) to specify a point in the event stream from which they want to begin reading events. You can specify the offset as a timestamp or as an offset value. Consumers are responsible for storing their own offset values outside of the Event Hubs service. Within a partition, each event includes an offset.

And also under Common consumer tasks - Read events:

As events are sent to the client, each event data instance contains important metadata such as the offset and sequence number that are used to facilitate checkpointing on the event sequence.

There do not seem to be any methods you can use to monitor the offset since you need to do this yourself.

Upvotes: 1

Related Questions