Setup webjob ServiceBusTriggers or queue names at runtime (without hard-coded attributes)?

Question

Is there any way to configure triggers without attributes? I cannot know the queue names ahead of time.

Let me explain my scenario here.. I have one service bus queue, and for various reasons (complicated duplicate-suppression business logic), the queue messages have to be processed one at a time, so I have ServiceBusConfiguration.OnMessageOptions.MaxConcurrentCalls set to 1. So processing a message holds up the whole queue until it is finished. Needless to say, this is suboptimal.

This 'one at a time' policy isn't so simple. The messages could be processed in parallel, they just have to be divided into groups (based on a field in message), say A and B. Group A can process its messages one at a time, and group B can process its own one at a time, etc. A and B are processed in parallel, all is good.

So I can create a queue for each group, A, B, C, ... etc. There are about 50 groups, so 50 queues.

I can create a queue for each, but how to make this work with the Azure Webjobs SDK? I don't want to copy-paste a method for each queue with a different ServiceBusTrigger for the SDK to discover, just to enforce one-at-a-time per queue/group, then update the code with another copy-paste whenever another group is needed. Fetching a list of queues at startup and tying to the function is preferable.

I have looked around and I don't see any way to do what I want. The ITypeLocator interface is pretty hard-set to look for attributes. I could probably abuse the INameResolver, but it seems like I'd still have to have a bunch of near-duplicate methods around. Could I somehow create what the SDK is looking for at startup/runtime?

(To be clear, I know how to use INameResolver to get queue name as at How to set Azure WebJob queue name at runtime? but though similar this isn't my problem. I want to setup triggers for multiple queues at startup for the same function to get the one-at-a-time per queue processing, without using the trigger attribute 50 times repeatedly. I figured I'd ask again since the SDK repo is fairly active and it's been a year..).

Or am I going about this all wrong? Being dumb? Missing something? Any advice on this dilemma would be welcome.

Answer 1

Based on my understanding, your needs seems to be building a message batch system in parallel. The @Thomas solution is good, but I think Azure Batch service with Table storage may be better and could be instead of the complex solution of ServiceBus queue + WebJobs with a trigger.

Using Azure Batch with Table storage, you can control the task creation and execute the task in parallel and at scale, even monitor these tasks, please refer to the tutorial to know how to.

Answer 2

The Azure Webjob Host discovers and indexes the functions with the ServiceBusTrigger attribute when it starts. So there is no way to set up the queues to trigger at the runtime.

The simpler solution for you is to create a long time running job and implement it manually:

public class Program
{
    private static void Main()
    {
        var host = new JobHost();
        host.CallAsync(typeof(Program).GetMethod("Process"));
        host.RunAndBlock();
    }

    [NoAutomaticTriggerAttribute]
    public static async Task Process(TextWriter log, CancellationToken token)
    {
        var connectionString = "myconnectionstring";
        // You can also get the queue name from app settings or azure table ??
        var queueNames = new[] {"queueA", "queueA" };
        var messagingFactory = MessagingFactory.CreateFromConnectionString(connectionString);
        foreach (var queueName in queueNames)
        {
            var receiver = messagingFactory.CreateMessageReceiver(queueName);
            receiver.OnMessage(message =>
            {
                try
                {
                    // do something
                    ....

                    // Complete the message
                    message.Complete();
                }
                catch (Exception ex)
                {
                    // Log the error
                    log.WriteLine(ex.ToString());

                    // Abandon the message so that it can be retry.
                    message.Abandon();
                }
            }, new OnMessageOptions() { MaxConcurrentCalls = 1});
        }

        // await until the job stop or restart
        await Task.Delay(Timeout.InfiniteTimeSpan, token);
    }
}

Otherwise, if you don't want to deal with multiple queues, you can have a look at azure servicebus topic/subscription and create SqlFilter to send your message to the right subscription.

Another option could be to create your own trigger: The azure webjob SDK provides extensibility points to create your own trigger binding :

• Binding Extensions Overview

Good Luck !