CODEWITHSUNDEEP
azure-functionsazure-eventgrid
amit agarwal
amit agarwal

Reputation: 83

Azure Event Trigger fires multiple times same file

I have a blob storage container where I have configured a Event-grid trigger (Blob Created). I am loading this blob storage files through Data factory and many times it will happen that many files may come up in this blob in one shot. May be we can take an example of 20 files.

The good news is my event-grid trigger is firing and the function app is called. However , I can see that sometimes for the same file the event-grid trigger is fired more than once.

Out of these 20 files there are few files which are very large say 300 MB but others are pretty small like in 3KBs. So my doubt is while this 300 MB is fired and it is still processing , parallelly it picks the same 300 MB file again (since it feels that it is still not read) and is saved in db multiple times which is not want I want.

Is Azure Event-grid would be the right approach for this scenario ?

Upvotes: 1

Views: 3556

Answers (2)

Pathrudu Majji
Pathrudu Majji

Reputation: 77

Once event grid trigger Azure Function for an event. It is excepting some reply from azure function in next 2 mins. If there is no response, Else Event grid will retry again.

Default value of retry for event grid is 30. Change it to 1.

Now even for large files that process for more than 2 mins, second duplicate trigger will not happen.

Upvotes: 1

Roman Kiss
Roman Kiss

Reputation: 8235

I can see, you opened a new thread for handling your problem which also is based on the Push-Push pattern for the event processing.

I do recommend to use the Push-Pull pattern, where the AEG will deliver an event to the queue storage and then based on the needs can be processed by Azure Queue Trigger function in the concurrent manner, see more details here.

Notes:

  1. The default timeout of the AF for Consumption plan is 5 minutes with a maximum 10 minutes. If this time is not enough for processing a large blob, you should use a Premium Plan where the default timeout is 30 minutes with a maximum time 60 minutes.

  2. In the case when you want to keep your business logic in the ADF pipeline, have a look at here how easy can be invoked the pipeline from the azure function.

Upvotes: 0

Related Questions