Architecture for Azure Functions App using an external REST API

Question

I want to try out Azure Functions with the following project:

• Triggered by time (every 30 minutes) my initial function1 puts some data in a queue1.
• This queue1 triggers another function2 that calls an external REST API, modifies the response and puts the results in another queue3.
• This queue3 starts another functions3 doing the rest.

My problem is that the REST API has a rate limiting. So if my function1 puts 100 items in the queue1 and the function2 is called 100 times parallel, my API calls will be blocked. I therefore need some kind of throttling.

How would you achieve that? I could tell function2 to wait a specific time and then add the item back to queue1, but since everything is parallel I might run in to a deadlock?

Thanks in advance for ideas!

Answer 1

is called 100 times parallel

You can limit this (to some extent) by configuring host.json:

• batchSize for Storage Queues

• maxConcurrentCalls for Service Bus

If that's not enough, you could do something more sofisticated:

• Function 1 knows how many items it has to process, so it could calculate the "ideal" distribution of those for the next 30 minutes

• When adding messages to queue1, it could set the time when the message should be picked up (ScheduledEnqueueTimeUtc for Service Bus or initialVisibilityDelay for CloudQueue)

• Function2 will be called "on schedule" which should prevent throttling, if the total amount of messages is not too high

Answer 2

I would recommend that you take a look at Azure Durable Functions here. The Durable Functions framework allows you to orchestrate complex workflows and manage state. In your example, you could use Durable Functions to work around the rate limiting issue