CODEWITHSUNDEEP
azureazure-webjobsazure-servicebus-queuesazure-webjobssdk
Bhanu Reddy
Bhanu Reddy

Reputation: 215

WebJob processing multiple messages at a time from queue

My WebJob is processing multiple messages from a queue which I don't want it to do. My queue has dependent messages which I want to process one after the other sequentially. I have tried with configuring "BatchSize" to 1 but no luck. It is still processing more than one message at a time.

Upvotes: 14

Views: 4407

Answers (4)

Informitics
Informitics

Reputation: 187

There is a reference to this here in the answer from @David Ebbo: Will an Azure Web job run on multiple instances?

What you are looking for is the setting:

{ "is_singleton": true }

Upvotes: 0

Reuel Ribeiro
Reuel Ribeiro

Reputation: 1479

If your messages have some order dependency, you are likely to have a pipeline of operations. I have a particular scenario where I use Azure Queues instead of ServiceBus, but I believe the following idea could apply the same:

I have an automation process which does: 1) create a database, 2) create a web app, 3) configure the web app, and 4) deploy code. For my case, things must be done in that order.

For that case, I have created one queue for each task: create-database-queue,create-webapp-queue,configure-web-app, and deploy-app-queue.

You are much safer to work that way because you isolate tasks. Once one process in one queue finishes, then it can be pushed to the next queue. Also, you could easily insert a new step into your process. For example, I can easily insert a new step 1.5) Restore database backup without a second thought.

Furthermore, you promote decoupling of responsibilities with individual queues.

Upvotes: 1

mathewc
mathewc

Reputation: 13558

The BatchSize setting (JobHostConfiguration.Queues.BatchSize) doesn't apply to ServiceBus queue processing, only to Azure Queues. To configure ServiceBus queue processing, use ServiceBusConfiguration. The knob you want to configure is ServiceBusConfiguration.MessageOptions.MaxConcurrentCalls. Set this to 1 to disable concurrent processing on a single instance (the default is 16):

JobHostConfiguration config = new JobHostConfiguration();
ServiceBusConfiguration serviceBusConfig = new ServiceBusConfiguration();
serviceBusConfig.MessageOptions.MaxConcurrentCalls = 1;
config.UseServiceBus(serviceBusConfig);

JobHost host = new JobHost(config);
host.RunAndBlock();

That will ensure that only a single message is processed at a time on a single instance. If your WebApp is scaled out, each scaled out instance would then be running in this mode, meaning you will have concurrent processing across instances. If you don't want that either, you can use SingletonAttribute to ensure only one instance of your function is running across instances. See the Singleton wiki page for more information.

Upvotes: 22

Pedro G. Dias
Pedro G. Dias

Reputation: 3222

What kind of queue it is depends largely on the success of what you're doing. I am not sure why you're getting issues with the queue batchsize - Azure Storage Queue gives you one message after another, as does also the service bus.

Azure WebJobs will scale out with your number of websites that you're hosting them on, which could give you concurrency issues, i.e. if you have a queue processor as a webjob running continously, then if your website scales to two instances, you WILL have two WebJobs also fighting for the same queue.

So if you require a single queue processor to operate, it would maybe be a better solution to move the webjob into a cloud service, where you have 100% control over the number of instances, clean separation from the WebSites and their instances.

Since you need to process each message serially, one after another, it would not make sense to scale out the number of processors, because they'd need to wait for the next message being processed anyways, so as long this is a requirement, and you MAY want to scale your WebSites, I would suggest CloudService as a better way to go.

Upvotes: -3

Related Questions