Service Bus Dead Letter Queue Processing In Azure Functions

Question

My environment.

• dotnet 6
• Azure Functions ( dotnet-isolated)
• Azure Service Bus

Now I have two functions with ServiceBusTrigger. One handle main queue message and another for handling dead letter queue message.

Scenario

• When main queue message process get failed ( for 2 times) then message moves to dead letter queue.
• Now it trigger the dead letter queue message and start retrying. Here it seems to be infinite loop. How to move message to another queue if it is failing at DLQ level. Is there any way to resubmit message to queue again.

Update-1

• Scenario is that there is something wrong with consumer that processing that message.
• Now when message first land in Main queue and then it process by consumer and something went wrong due to some dependency issue with consumer. So after some retry message will land in DLQ.
• Now DLQ use same business code to process message but it is from different function. As there is something wrong with dependency so again message will get fail. Now in DLQ it processing infinite. I want to move this message to some other queue and from there some manual intervention required to process that message further.
• In high traffic application it is possible that before above decision has made there are many message in queue/DLQ.

Answer 1

The process which picks up the messages from the DLQ should not be something that has chances of failing so often. The fact that messages in the DLQ are going into loop itself suggests that something is really wrong here.

Can you elaborate what your process is doing and why is it failing? Are you triggering the same functionality from the main queue as you are from the DLQ? That would not be wise. If you can look at the root cause of the failures, you will have less messages landing up in the DLQ to handle.

Think of it this way. Would you need the additional queue to retry and then push it to another queue? How many more queues would you need before you finally move it to a dead letter queue? What would you do with the dead letter queue then?

Anyhow, Service bus will not automatically move messages from the DLQ to another queue. But you can trigger your own process to write it into another queue if you really need to. Service bus triggered Azure functions is perfect to do such work.

To break that infinite loop from the DLQ, add a check for the delivery count value from the Brokered properties. If this value is greater than a certain threshold, then write the message into another queue. The delivery count is incremented each time the message is picked up.

An example of this in .NET is shown below. I'm simulating a failure to illustrate.

using System;
using System.Threading.Tasks;
using Microsoft.Toolkit.Services;

public static void Run(string myQueueItem, Int32 deliveryCount, ILogger log)
{
    log.LogInformation($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
    log.LogInformation($"Delivery count : {deliveryCount}");
    if (deliveryCount < 3)
    {
        log.LogInformation("Simulating failure");
        throw new TooManyRequestsException();
    }
    else
    {
        log.LogInformation($"Delivery count is now {deliveryCount}. Time to write this message into another queue.");
    } 
}

On my 3rd retry, I would like to send my message to another queue. After sending a test message to my service bus, I get the following output.

2022-02-22T03:05:35.281 [Information] Executing 'Functions.ServiceBusQueueTrigger1' (Reason='(null)', Id=eed59015-f100-42b4-b0bf-57a9a6ab226f)
2022-02-22T03:05:35.281 [Information] Trigger Details: MessageId: 86f8250e82fb42deb88e6b91244ca2b7, SequenceNumber: 3, DeliveryCount: 1, EnqueuedTimeUtc: 2022-02-22T03:05:35.2980000Z, LockedUntilUtc: 2022-02-22T03:06:05.2980000Z, SessionId: (null)
2022-02-22T03:05:35.531 [Information] C# ServiceBus queue trigger function processed message: my new message
2022-02-22T03:05:35.531 [Information] Delivery count : 1
2022-02-22T03:05:35.532 [Information] Simulating failure
2022-02-22T03:05:35.557 [Error] Executed 'Functions.ServiceBusQueueTrigger1' (Failed, Id=eed59015-f100-42b4-b0bf-57a9a6ab226f, Duration=253ms)Exception of type 'Microsoft.Toolkit.Services.TooManyRequestsException' was thrown.
2022-02-22T03:05:35.644 [Information] Executing 'Functions.ServiceBusQueueTrigger1' (Reason='(null)', Id=78d2df93-11f6-412b-9ae7-880ac3bf4213)
2022-02-22T03:05:35.645 [Information] Trigger Details: MessageId: 86f8250e82fb42deb88e6b91244ca2b7, SequenceNumber: 3, DeliveryCount: 2, EnqueuedTimeUtc: 2022-02-22T03:05:35.2980000Z, LockedUntilUtc: 2022-02-22T03:06:05.6730000Z, SessionId: (null)
2022-02-22T03:05:35.645 [Information] C# ServiceBus queue trigger function processed message: my new message
2022-02-22T03:05:35.646 [Information] Delivery count : 2
2022-02-22T03:05:35.646 [Information] Simulating failure
2022-02-22T03:05:35.674 [Error] Executed 'Functions.ServiceBusQueueTrigger1' (Failed, Id=78d2df93-11f6-412b-9ae7-880ac3bf4213, Duration=2ms)Exception of type 'Microsoft.Toolkit.Services.TooManyRequestsException' was thrown.
2022-02-22T03:05:35.782 [Information] Executing 'Functions.ServiceBusQueueTrigger1' (Reason='(null)', Id=b856141d-d891-4edb-a910-7e53b40b96a0)
2022-02-22T03:05:35.783 [Information] Trigger Details: MessageId: 86f8250e82fb42deb88e6b91244ca2b7, SequenceNumber: 3, DeliveryCount: 3, EnqueuedTimeUtc: 2022-02-22T03:05:35.2980000Z, LockedUntilUtc: 2022-02-22T03:06:05.8130000Z, SessionId: (null)
2022-02-22T03:05:35.783 [Information] C# ServiceBus queue trigger function processed message: my new message
2022-02-22T03:05:35.784 [Information] Delivery count : 3
2022-02-22T03:05:35.784 [Information] Delivery count is now 3. Time to write this message into another queue.**
2022-02-22T03:05:35.784 [Information] Executed 'Functions.ServiceBusQueueTrigger1' (Succeeded, Id=b856141d-d891-4edb-a910-7e53b40b96a0, Duration=2ms)