CODEWITHSUNDEEP
azure-webjobsazure-queues
daxu
daxu

Reputation: 4084

Is it possible to bypass the 32 limit when using azure webjob and queue?

I am learning azure webjob and queue, in the company I work for, every night, we calculate like 7 million records. The calculation is quite simple, but the quantity is quite big (as there around 7 million records in db).

As the company is moving to Azure, I am wondering can azure queue be used to handle this? In theory, I can create some code to populate the queue (as there is a 64kb message size limit, I guess I can put like 10 records in one message, so total message will be like 700k?). Then I will have webjob that triggered by the queue and do the calculation.

However, according to here,there is a max limit of 32 in message retrieving.

Does this mean that there will be a maximum of 32 webjob triggered concurrently?If that is the case, 700000/32 = 21875, which means it will take quite a while to finish.

Is there a way to trigger more webjob to run concurrently other than the 32 limit?

Upvotes: 1

Views: 597

Answers (1)

David Ebbo
David Ebbo

Reputation: 43203

You can get much higher concurrency by combining BatchSize and NewBatchThreshold. The best way to look at it is that the concurrency limit is the sum of the two flags. So if you leave BatchSize at 32 and set NewBatchThreshold to 100, the concurrency limit will be 132.

See https://github.com/Azure/azure-webjobs-sdk/issues/628 more for details.

Upvotes: 1

Related Questions