CODEWITHSUNDEEP
azureazureservicebusazure-eventhubazure-storage-queues
Robert Bengtsson
Robert Bengtsson

Reputation: 194

Solution for delaying events for N days

We're currently writing an application in Microsoft Azure and we're planning to use Event Hubs to handle processing of real time events.

However, after an initial processing we will have to delay further processing of the events for N number of days. The process will work like this:

Event triggered -> Place event in Event Hub -> Event gets fetched from Event Hub and processed -> Event should be delay for X days -> Event gets' further processed (two last steps might be a loop)

How can we achieve this delay of further event processing without using polling or similar strategies. One idea is to use Azure Queues and their visibility timeout, but 7 days is the supported maximum according to the documentation and our business demands are in the 1-3 months maximum range. Number of events in our system should be max 10k per day.

Any ideas would be appreciated, thanks!

Upvotes: 1

Views: 1160

Answers (1)

Sreeram Garlapati
Sreeram Garlapati

Reputation: 4993

As you already mentioned - EventHubs supports only 7 days window of data to be retained.

Event Hubs are typically used as real-time telemetry data pipe-lines where data seek performance is critical. For 99.9% usecases/scenarios our users typically require last couple of hours, if not seconds.

However, after the real-time processing is over, and If you still need to re-analyze the data after a while, for ex: run a Hadoop job on last months data - our seek pattern & store are not optimized for it. We recommend to forward the messages to other data archival stores which are specialized for big-data queries.

As - data archival is an ask that most of our customers naturally look for - we are releasing a new feature which automatically archives the data in AVRO format into Azure storage.

Upvotes: 1

Related Questions