CODEWITHSUNDEEP
c#azureazure-webjobsazure-queues
Pure.Krome
Pure.Krome

Reputation: 86947

How to process lots of messages on an Azure Queue with a WebJob?

I have a pretty simple queue which happens to have heaps of messages on it (by design). Heaps can be "thousands" here.

Right now I'm playing around with using Azure Web Jobs with a Queue Trigger to process the messages. Works fine.

I'm worried about performance though. Let's assume my method that processes the message takes 1 sec. With so many messages, this all adds up.

I know I can manually POP a number of messages at the same time, then parallel process them .. but I'm not sure how we do this with web jobs?

I'm assuming the solution is to scale out? which means I would create 25 instances of the webjob? Or is there a better way where I can trigger on a message but pop 25 or so messages at once, then parallel them myself.

NOTE: Most of the delay is I/O (ie. a REST call to a 3rd party). not CPU.

I'm thinking -> create 25 tasks and await Task.WhenAll(tasks); to process all the data that I get back.

So, what are my options?

NOTE #2: If the solution is scale out .. then I also need to make sure that my web job project only has one function in it, right? otherwise all the functions (read: triggers, etc) will also all be scaled out.

Upvotes: 1

Views: 861

Answers (1)

Abhay Saraf
Abhay Saraf

Reputation: 1212

Azure WebJobs have a default configuration of processing 16 messages in parallel and this number is configurable. The WebJobs framework internally creates 16 (or the configured MaxDequeueCount) copies of the your function and runs them in parallel.

Moreover, you can launch multiple instances, i.e. scale out, of the Azure App Service/Website hosting the WebJob, subject to maximum of 20 instances. However, scaling out instances (unlike parallel Dequeue above) has a pricing impact so please check on that.

Thus theoretically you could be processing 24*20 = 480 messages in parallel by configuration alone on a standard WebJob function without any custom code.

Upvotes: 3

Related Questions