CODEWITHSUNDEEP
azure-functionsazureservicebus
Mike Devlin
Mike Devlin

Reputation: 51

Azure Functions: Service Bus Messages Processed Twice

We have several Azure Function (1.0) apps with a ServiceBus trigger. We are experiencing a sporadic problem where messages are picked up and processed twice, which causes huge problems.

Our sample code looks like this:

[FunctionName("CreateFile")]
public static async Task Run([ServiceBusTrigger(CreateFileTask.topic, "default", Connection = "Queue.ConnectionString")]string mySbMsg, ILogger log)
{
...

The most recent time this occurred, the message was retrieved and started processing at 7:00pm. At 7:03pm, the message was again retrieved and processed in parallel. Both processes finished successfully, which resulted in the creation of two files.

This problem doesn't happen frequently (maybe once every two months), but it is a huge headache when it does.

How can we prevent this from occurring in the future?

Upvotes: 1

Views: 1940

Answers (2)

Mike Devlin
Mike Devlin

Reputation: 51

For anyone who encounters this issue in the future: this problem can only be solved within code, not the Message Bus or Azure Function settings.

Specifically: Microsoft requires that the application listening to the Service Bus check the MessageId for duplicates before you begin processing the message. Maybe this is common knowledge; we didn't know that, and I didn't see it in the documentation anywhere.

After I created this Stack Overflow issue, we changed the settings on the Service Bus, reducing the MessageDeliveryCount to 1 and MessageLockDuration to 5 minutes. However, this issue still occurred a few days later. We spoke to Azure Support following this second incident. They informed us that Service Bus guarantees each message will be delivered at least once, but not only once. If Service Bus has any reason to "suspect" that a message was not properly delivered, it will attempt a re-delivery. Therefore, the code that reads the message must perform its own duplicate checking.

Upvotes: 4

Sean Feldman
Sean Feldman

Reputation: 25994

If the original message was processed at 7:00pm and duplicate processing took place in parallel at 7:03pm, your lock duration might be shorter than the maximum message processing time. As a result of that, a message could and would be processed more than once. Except one execution would log a failure to complete the message when the other would succeed.

Upvotes: 2

Related Questions