CODEWITHSUNDEEP
azureservicebusmasstransit.net-core-3.1azure-servicebus-queues
Joel
Joel

Reputation: 8998

Publishing messages on Azure Service Bus using MassTransit is intermittently slow

This might not be a MassTransit question per se, but I'm trying to debug what's going on.

We are using MassTransit (7.1.8) with Azure Service Bus. Most of our messages are published just fine in a timely manner, but sometimes messages take a really long time to publish. And I don't understand why.

Our 4 (premium) ASB instances are under load, but CPU/Memory isn't above ~70% which I guess shouldn't be alarming.

Here's a snippet from our logs:

03 Jun 2021 13:41:55.642 IBusControl.PublishEvents() with 1 took 00:00:36.1610001
03 Jun 2021 13:41:55.642 SEND sb://<namespace>.servicebus.windows.net/events/...
03 Jun 2021 13:41:19.481 IBusControl.PublishEvents() called with 1 items.

And the corresponding code:

logger.LogDebug("IBusControl.PublishEvents() called with {count} items.", count);
var stopWatch = Stopwatch.StartNew();
var tasks = updatedItems.Select(x => bus.Publish<ISomeEvent>(new SomeEvent(x, x.Parameters)));
await Task.WhenAll(tasks);
stopWatch.Stop();
logger.LogDebug("IBusControl.PublishEvents() with {count} took {elapsed}", count, stopWatch.Elapsed.ToString());

So I'm trying to figure out why it takes >30 seconds between that we call Publish and when MassTransit logs the SEND entry.

Not really sure where to look? Any help would be appreciated.

EDIT

One interesting thing that I've noticed is that the time it takes is often around ~10 seconds, no matter how many messages there are to publish 🤯

enter image description here

Upvotes: 2

Views: 850

Answers (1)

Chris Patterson
Chris Patterson

Reputation: 33540

As explained in the comments above, Azure Service Bus was throttling the namespace, forcing the Azure SDK to fail and automatically retry the operation after a ten second delay.

The request was terminated because the namespace is being throttled. Error code : 50002. Please wait 10 seconds and try again

The failure and subsequent retry are handled entirely within the Azure SDK components, and not visible to MassTransit.

Upvotes: 2

Related Questions