Chris K
Chris K

Reputation: 560

Why does Azure queue triggered flex consumption function keep re-triggering?

I have an Azure function which generates a PDF report. The reports can be thousands of pages long and contain tens of thousands of images.

Background to the problem:

It is a queue-triggered function. I deployed it initially on the consumption plan, and this works fine except for larger reports, where the 10 minute runtime-limit isn't enough.

I've run the function locally with large reports and they may take around 25 minutes to complete, but they do complete without issue.

When published to a flex-consumption plan, messages are removed from the queue and reports created, but the message seems to then be put back on the queue, and the function triggers again. This happens five times before every request then ends up on the poison queue.

I have no failed requests - all are logged as successfully completed.

The queue triggered-function is defined as:

[Function(nameof(PdfConversionTrigger))]
public async Task Run([QueueTrigger("pdfcreation", Connection = "<connection string>")] QueueMessage message)
{
    _logger.Log(LogLevel.Information, message.ToString());
    QueuePdfDTO? queueMessage = JsonConvert.DeserializeObject<QueuePdfDTO>(message.MessageText);
    if (queueMessage is null) return;
    
    //Begin processing the PDF...
    await CreatePDF(queueMessage);
}

My hosts.json file for the function is:

{
"functionTimeout": "00:50:00",
"version": "2.0",
"logging": {
    "applicationInsights": {
        "samplingSettings": {
            "isEnabled": true,
            "excludedTypes": "Request"
        },
        "enableLiveMetricsFilters": true
    }
},
"extensions": {
    "queues": {
        "batchSize": 1,
        "messageEncoding": "none"
    }
}

I don't know where to start looking to fix this, so any pointers would be gratefully received. I have seen that there are durable functions, but the closest use-case I could see was for HTTP triggered functions.

I am confident that the function completes successfully, as the final steps of the function update the status of a database entity and save the PDF to blob storage, both of which happen. However, a couple of minutes later, the job starts up again, until five cycles have completed and the job ends on the poison queue.

For what it's worth, on a job which took roughly 6 seconds @ 2GB (so ~12MBs consumption) everything worked fine. On a job which takes approx 2 minutes (~120MBs) the job runs to completion, and then 7-8 minutes later, it repeats, until five repeats at which point it adds the message to the poison queue. I have verified that each invocation runs to completion, and that no other messages are being added to the queue.

Upvotes: 1

Views: 47

Answers (0)

Related Questions