CODEWITHSUNDEEP
c#azuretask-parallel-libraryazure-worker-roles
LivingOnACloud
LivingOnACloud

Reputation: 1261

How do I make an Azure worker role scale if the thread is busy waiting?

I hope the below explains my problem.

I have a worker role which is running in a while(true) loop. This worker is taking messages off a queue and processing them. It never finishes, just keeps checking a queue and processing messages.

This works until I get a message which requires me to do something which takes time. An example might be requesting information from a service. Hypothetically let us say this request takes 10 seconds. Now my worker role is in the middle of my while loop waiting for this response. Meanwhile the message queue is filling up.

What I want is a solution which will either stop or mitigate this from happening.

My current real world solution is that I have a worker role which uses the task parallel library to create a number of message processors which run in parallel and take from the queue. But still this problem might happen if all threads running my message processors get stuck waiting for a service.

I know that I should use async requests for services but there are other examples where async will not help. I would rather not go into the details.

So my plan right now is to modify my current architecture and create a message processing service. So I will consume the message queue in the current solution and only use the worker role to delegate the work to my web service (asynchroniously).

This worker role will now not have the situation of threads getting stuck waiting on something.

But I'm not happy with this. What if my worker role which uses the task parallel library to create a number of queue processors isn't fast enough. I don't think the CPU will even stress because the slow down will be in retrieving the message and sending it to a web service, both not CPU intensive.

So I need MORE queue processors which will have to be created by spawning instances of my queue processor based off how busy my queue is. So when the queue has a lot of items in it, Azure will create a new instance to help me process them.

I'm not confident about this solution. I feel like this should be easier. Why am I now creating a web service to consume messages which are sent from a worker role which in turn is scaled based off my message queue.

Does my solution make sense or are there alternatives?

Upvotes: 2

Views: 663

Answers (1)

James
James

Reputation: 82096

Look at the reason you need to scale

Hypothetically let us say this request takes 10 seconds. Now my worker role is in the middle of my while loop waiting for this response. Meanwhile the message queue is filling up.

The problem is not so much that a worker takes a long time but rather the queue gets backed up - that's exactly why Azure offers scaling options which include Queue size.

Additionally, if your services are CPU intensive then scaling out is only going to spread the problem onto multiple instances - it's only a matter of time before you hit the same bottleneck.

Ultimately, you need to make a decision on how long is too long, and then figure out whether the queue would process quicker with more help (scaling out) or more muscle (scaling up), or both.

Upvotes: 2

Related Questions