Azure Function Event Hub Trigger one batch at a time

Question

I am working on a project which gets change notifications through Event Hubs(https://learn.microsoft.com/en-us/graph/change-notifications-delivery).

My Azure Function(Event Receiver) receives the events in batches. After receiving each batch, it processes the data (does a bit of work). The second batch updates the value set by the first batch.

The Event Hub has 3 partitions.

The problem arises when multiples batches arrives at the same time. . The second batch comes in before the first batch has updated the value.

Is there a way to receive only one batch at a time?

I am not bothered about the order of batches, but I want to get the next batch trigger only after the first batch is processed.

Thanks in advance.

Answer 1

Thank You BrunoLucas-9843 and MughundhanRaveendran-MSFT for your valuable explanation and few workarounds. Posting your suggestions as an answer to help other community members.

• Check out this article regarding the configuration of maxBatchSize to pull a batch of 1 message for the function and the better way to reduce cost when passing single event vs ordered batch.

Is there a way to make the event hub trigger wait until the first batch is processed?

• As soon you send the event and it hits the hub, the trigger function will pull that. There is also azure function parallelism that makes things more complicated.

• Best I can think is to write a method like this: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-dotnet-standard-getstarted-send#receive-events and use the Timer trigger to pull it every "X" minutes being X the average time it takes to process one whole batch (plus some extra time to make sure it clear it all).

I see that logic apps triggers for azure hub can pull a fixed number of events in one pull and wait:
https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-azure-event-hubs#trigger-polling-behavior

Additionally you can try to set the cardinality to one in the function.json and test it at your end. https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger?tabs=csharp#configuration

Would it be possible to achieve the sequential processing using a Durable Function?

Yes, it is possible with durable and normal azure functions as well by leveraging the above suggested methods. With durable functins, you can use functions chaining : https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=csharp#chaining.

In durable functions, there is orchestrator and activity functions, if one of the activity function fails it will return error response to the orchestrator. The entire instance will not fail, you will be able to handle the exception