CODEWITHSUNDEEP
azureazure-functionsazure-eventhub
Stanislas
Stanislas

Reputation: 2020

Azure Event Hubs multiple consumers within one application

Context

I have an Azure Functions application deployed in Azure that needs to process events from an Event Hub.
Within this application I want some events to be processed for multiple independent features.

Requirements

What I want to avoid is that if one feature fails, that all other features would fail too.
Additionally, I'd like to have an easy way of retrying an event for a specific feature.

Research

After reading the documentation and browsing the internet I've come across Consumer Groups, which indicate that they can be used for independent parallel processing.

However, I get the feeling that Consumer Groups aren't meant for my case and should be seen more as 1 consumer group per application rather than 1 consumer group per feature.
Am I correct in this assumption?

After searching more, I haven't really come across anyone tackling this problem, while it feels to me as if it should be quite common.
How would this normally be achieved?

Potential solution?

One thing I have been considering is to have a single Azure Function that receives all events.
When an event is received, this Azure Function would check who is interested in the event and would then per feature send the same event to a different Event Hub with an indication of which feature it should be processed for (partitioned by the feature).
This would then allow the events to be processed independently per feature.

Is this a "normal" way of processing events via Event Hub?
Are there any pitfalls that I'm missing?

Upvotes: 0

Views: 2711

Answers (1)

Jesse Squire
Jesse Squire

Reputation: 7875

If I understand your scenario correctly, it sounds like you have multiple independent features, each of which would like a unique view of events flowing through the system. Each feature should process independently and be isolated from other features to prevent cascading failures. Assuming that's correct, I'd first start with a question:

Are you seeing events flow through at a high enough scale that Event Hubs makes sense? To me, it seems like a Service Bus topic with a subscription for each feature may be the more natural pattern. Unless you're processing a high volume of events consistently, it may be worth exploring.

If you're convinced that Event Hubs is the right fit, then I'd consider the following skeleton to start exploring from, assuming that you're also sticking with Functions:

  • One consumer group per feature. (note: There's a limit of 20 consumer groups for a standard instance; if you have more than 20 features, you would need to either consider sharding among multiple Event Hub instances or using a dedicated tier)

  • One Function per feature, bound to the associated consumer group

The consumer groups will allow you to have the isolation that you're looking for, with features being independent and being able to recover/reprocess without impacting other features. It will also allow you to scale horizontally much more efficiently with Functions managing.

I wouldn't advise using a single function for all instances. You'll end up having a lot more complexity that you have to manage directly in order to track the state of each feature independently, as checkpoints and state for reading the partition exist at the consumer group level. You would also be doing considerably more work in a single instance, making it more difficult to scale effectively. The best you'd be able to do would be one instance for each partition of the Event Hub.

Upvotes: 1

Related Questions