setting a function timeout on azure functions

Question

I was given a task to write a http trigger function which will take some blob storage on azure, read all of its blobs (JSON data) and based on some of the values inside the JSON copy the blobs into specific sub directories. Because of the workload (70k of JSONs) and the number of requests the function needs to make to read and later copy the JSON data I kept running into timeouts, so for my local implementation (it was easier to manage timeouts in local development) I just added in host.json functionTimeout field and set it to 23:50:00 as it was around maximum a local run can support (I think its 24h). I kept running it for couple of days and all of the JSONs were sorted into subdirectories. Now I want to publish this function to go on azure and to rerun it to make sure that it also works. I changed the app service plan plan to be dedicated resource (Basic B3), but I keep getting timed out after 300s, also added AzureFunctionsJobHost__functionTimeout app settings, but I still keep timing out.

The question is, what am I doing wrong here, should I switch to a durable function and use Consumption plan, or change some app setting or host.json, or something third, basically how to get my function (Python 3.10) to stop timing me out.

Answer 1

Firstly, I'm assuming it's important that you start this process from an HttpTrigger e.g. because an external process is deciding when this should run. If not, then do consider Donny Kwitty's suggestion in the comments.

You're right that it's not acceptable for an Azure Function to take a long time to run. Even if you opt for a higher tier plan to get a longer timeout, reliance on that is probably a sign that you should re-think your approach or pick a different platform.

Additional to the general Azure Function timeout, there's a further restriction on HttpTrigger - see https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#timeout And regardless of whether you're using Azure Functions or any other technology, you just can't have long-lived HTTP endpoints. There's a good chance the system that's calling your HttpTrigger will close the connection if it's taking that long.

The good news is that you have a few options in the Azure Functions ecosystem. To help decide, answer the following questions:

1. Is it vital that I only process only one Blob at a time?
2. Does the HTTP caller need to know when all the Blobs have finished processing?

If the answer to either of these is "yes" then you probably want Durable Functions. If not, you could consider Azure Queues. I'll describe each below.

Durable Functions option

This will let you loop through them one by one. You'd have a Durable Orchestration, which would first call a Durable Task to fetch all the Blob URLs at the start. The Orchestration would then loop through them awaiting another Durable Task for each Blob.

This avoids an Azure Functions timeout because the Orchestration effectively exits each time it awaits the next Durable Task.

For greater efficiency, you should consider grouping the Blob URLs into small batches, and processing a whole batch in each Durable Task. This would mitigate some of the fixed overhead of calling a Durable Task.

For added robustness against the caller accidentally starting another instance before the first has finished, you can use Durable Locks to quickly exit when a second Orchestration tries to run. See https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-entities?tabs=function-based%2Cin-process%2Cpython-v2&pivots=csharp#critical-section-behavior

Note: even with Durable Functions, you can't just have an HttpTrigger that waits indefinitely until all the Blobs are processed. The Durable Orchestration doesn't go in the the HttpTrigger function itself. Instead, the HttpTrigger will trigger the Orchestration and return immediately. See https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=in-process%2Cnodejs-v3%2Cv1-model&pivots=csharp#async-http

Azure Queues option

Your HttpTrigger could fetch a list of the Blob URLs, and then enqueue each Blob URL as an Azure Queue item.

You'd then have a QueueTrigger function that picks up the Blob URL and processes it. Similar to the Durable approach, you could optimise this by batching several Blob URLs into one Queue item, so it's not firing quite so many QueueTrigger instances.

Final thoughts

Do be aware that the above patterns need extra thought about making them resilient. They're harder than just having a single method that safely loops around a list. Things can happen such as:

• Queue Triggers can sometimes run twice with the same input data (sometimes even at the same time) which might cause your Azure Storage to throw an exception because two processes are trying to modify the same file.
• Blobs may get processed in an unexpected order, e.g. if there's a delay processing one Blob and the Queue Trigger for the next one executes.
• If your client calls the HttpTrigger endpoint a second time, before you've finished processing the first set of blobs, you might either:
  • be processing the same Blob at the same time in two instances, causing one or both to fail
  • find some Blobs have gone missing in the second instance, causing it to throw an exception and pointlessly generate a Retry in the Azure Queue, hurting performance

Make sure you have proper exception handling for the likely Azure Storage exceptions. And have some good logging to fall back on when things go wrong!