CODEWITHSUNDEEP
c#azureconcurrencyazure-functions
JoeTomks
JoeTomks

Reputation: 3276

Azure Function App v4 Blob Trigger Concurrency

I've been looking around online for the last couple of days and can't find a difinitive answer to my problem.

I'm building an enterprise scale solution in Azure, fairly simple, there are two function apps, one that processes the upload of a CSV file to blob storage and replies with a 200 OK response. This works fine and scales upto silly amounts of concurrent requests coming in over our Application Gateway.

There's a second function app that then, using a Blob Trigger, starts to pick the CSV files up from blob storage and process them ready to be inserted into SQL. This is the bit of the solution I'm struggling to understand what Microsofts guidance really is. For example if we recieve let's say 100 csv files concurrently, ideally I want as many as possible to be picked up by this function app and processed, they can all be processed concurrently as there's no dependancy between any of the CSV data.

What I'm seeing right now is maybe 6-8 being processed at a time, the function app is a consumption based one, and it appears to work at a higher level of concurrency than a function app on a premium plan based on my tests in Azure. So I think I'm likely missing something. If anyone has dealt with this particular scenario before and has some advice on settings, or configurations that I can tweak or adjust to get the desired level of scaling then I'd be very greatful.

Upvotes: 1

Views: 382

Answers (1)

Peter Bons
Peter Bons

Reputation: 29780

I wouldn't use a blob triggered azure function as it involves polling to determine what blobs to process. From the docs:

If you require faster or more reliable blob processing, consider creating a queue message when you create the blob. Then use a queue trigger instead of a blob trigger to process the blob. Another option is to use Event Grid; see the tutorial Automate resizing uploaded images using Event Grid.

I would go for an event grid based approach. That service scales very well.

Upvotes: 2

Related Questions