CODEWITHSUNDEEP
azure-functionsazure-eventhub
user3012708
user3012708

Reputation: 946

Changing application that processes Event Hub stream results in starting from epoch 0, why?

I have two Function apps that are processing an Event Hub, they're both using the same consumer group.

Only one function in each Function app actually interacts with the Event Hub.

I've disabled App1's function that interacts with the Event Hub and enabled App2's function. This means that the functions that interact with the Event Hub in each Function app never run at the same time.

What I find though, is that upon disabling App1, and enabling App2's function, App2 will always read from epoch 0 of the Event Hub. This means I re-process a large chunk of data every time I switch between App1 and App2's function.

How do I get them to share the state so that they're both running from a shared epoch? (which I know is recorded in the Event Hub's attached storage).

I'm wondering if I need to completely shut down App1 - not just disable the function? Sounds like there's a lease problem, but I don't quite understand it.

And for the record - I will not be switching between these Function Apps in the future, App2 will replace App1, but I can't have it re-processing data. (Though the data is idempotent, it would result in a large hit to traffic and performance, along with storage costs).

EDIT: Previous app was Functions 1.x and .Net Framework. New app is Functions 4.x and .Net 6. So lots of changes, but same functionality implemented.

Upvotes: 1

Views: 195

Answers (1)

juunas
juunas

Reputation: 58743

Based on the discussion in the comments, the reason that the checkpoints were not used was that the way the Functions stored information about checkpoints was different.

The older SDK stores the epoch and offset inside the blob content, while the newer one uses blob metadata. Putting those values in metadata before starting the newer function seems to have fixed the issue.

Upvotes: 1

Related Questions