CODEWITHSUNDEEP
c#azureazure-storageazure-storage-queues
JohnB
JohnB

Reputation: 59

Azure queue throttling error with few requests (StorageException: The server is busy)

Hi,

I use a queue storage on Azure. I have 4 programs which make each 1 request/second (GetMessagesAsync request). So I make 4 request/seconds.

But sometimes, there is throttling errors that results on this exception :

System.AggregateException: One or more errors occurred. (The server is busy.) ---> Microsoft.WindowsAzure.Storage.StorageException: The server is busy.\r\n at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.d__4`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.<>c__DisplayClass83_0.<b__0>d.MoveNext()\r\n

I don't know why I get this error given that Azure target throughput for single queue is up to 2000 messages per second.

Additional informations :

My function where I make the get :

protected async Task<IEnumerable<CloudQueueMessage>> GetAndDequeueMessagesAsync(int numberOfMessagesToRetrived = 1) {
            try
            {
                IEnumerable<CloudQueueMessage> messages = await GetMessagesAsync(numberOfMessagesToRetrived);
                foreach (CloudQueueMessage queueMessage in messages)
                {
                    await DequeueMessageAsync(queueMessage);
                }
                return messages;
            }
            catch (Exception e)
            {                
                return new List<CloudQueueMessage>();
            }            
        }

Thanks

Upvotes: 1

Views: 2261

Answers (1)

Brando Zhang
Brando Zhang

Reputation: 27997

According to your description, I guess the reason is the storage service moving partitions to improve load balancing.

As far as I know, the azure storage partition server will not just host one partition(queue\table partition-key), it will host many partitions.

If the partition server receive too many request, azure storage will automatic improve load balancing.

It will move some partitions to another partition server. If you send reqeust when moving the partition it will return 503 error.

More details about how the azure storage improve load balancing. You could refer to this azure storage SOSP article's 5.5.1 Load Balance Operation Details .

I suggest you could implement retry policies using the Storage Client Library to keep your program working well.

More details, you could refer to below article: Microsoft.WindowsAzure.Storage.RetryPolicies Namespace

Upvotes: 1

Related Questions