Azure Function QueueTrigger not receiving cancellation requests

Question

I have a .NET 8 Azure Function v4 QueueTrigger with the dotnet-isolated runtime. In production, we have run into an issue where the function application will shutdown mid-execution, leaving the application in an invalid state:

2024-06-05 02:31:15.782 [Information] [redacted legitimate activity, 1/250]
2024-06-05 02:31:15.860 [Information] [redacted legitimate activity, 2/250]
2024-06-05 02:31:15.866 [Information] [redacted legitimate activity, 3/250]
2024-06-05 02:31:16.233 [Information] [Microsoft.Hosting.Lifetime] Application is shutting down...
2024-06-05 02:31:31.133 [Information] [Microsoft.Hosting.Lifetime] Application started. Press Ctrl+C to shut down.

So I added a CancellationToken to the function method signature hoping that I would be able to observe cancellationToken.IsCancellationRequested and shutdown the application gracefully, but its value is always false:

[Function(nameof(Function1))]
public async Task Run([QueueTrigger("test-queue")] QueueMessage message, CancellationToken cancellationToken)
{
    _logger.LogInformation($"C# Queue trigger function processed: {message.MessageText}");
    _logger.LogInformation("cancellationToken == CancellationToken.None? {IsEqual}", cancellationToken == CancellationToken.None);

    cancellationToken.Register(() =>
    {
        Console.WriteLine("CancellationToken Register callback invoked.");
    });

    foreach (var ix in Enumerable.Range(start: 0, count: 1_000))
    {
        await Task.Delay(1_000);

        if (cancellationToken.IsCancellationRequested)
        {
            _logger.LogInformation("Cancellation requested!");
            return;
        }
        else
        {
            _logger.LogInformation("Cancellation not requested.");
        }
    }
}

When running locally in Visual Studio, I'm trying to simulate this shutdown with CTRL+C. When running in Azure, I'm trying to simulate this shutdown by disabling the function while it's in mid-execution. Neither results in the CancellationToken having a IsCancellationRequested property value of true.

What am I doing wrong?

Reproduction in this GitHub repo

Thank you.

ETA 2024-06-13: When running in-process, the CancellationToken works as expected. When running in dotnet-isolated, it does not behave as expected. GitHub issue

Answer 1

In your code, CancellationToken is requested when the loop is completed. Your loop runs for 1000 times, as the count is set to 1000. Based on this, your loop will execute 1000 times before the CancellationToken is set to true.

I am using a count of 10, as you can see, the loop executes 10 times, and then the CancellationToken is set to true.

Code :

Function.cs :

using Azure.Storage.Queues.Models;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;

namespace AzureFunctionQueueTriggerCancellationTokenTest;

public class Function1
{
    private readonly ILogger<Function1> _logger;

    public Function1(ILogger<Function1> logger)
    {
        _logger = logger;
    }
    [Function(nameof(Function1))]
    public async Task Run([QueueTrigger("<queueName>", Connection = "AzureStr")] QueueMessage message, CancellationToken cancellationToken)
    {
        _logger.LogInformation($"C# Queue trigger function processed: {message.MessageText}");
        _logger.LogInformation("cancellationToken == CancellationToken.None? {IsEqual}", cancellationToken == CancellationToken.None);

        cancellationToken.Register(() =>
        {
            Console.WriteLine("CancellationToken Register callback invoked.");
        });

        foreach (var ix in Enumerable.Range(start: 0, count: 10))
        {
            await Task.Delay(1_000);

            if (cancellationToken.IsCancellationRequested)
            {
                _logger.LogInformation("Cancellation requested!");
                return;
            }
            else
            {
                _logger.LogInformation("Cancellation not requested.");
            }
        }
    }
}

Program.cs :

using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
        services.Configure<LoggerFilterOptions>(options => options.Rules.Clear());
    })
    .Build();

host.Run();

local.settings.json :

{
    "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
    "AzureStr": "<storage_conneString>"
  }
}

Output :

Azure Portal :

After deployment, I added the following connection in the Azure Function App's Environment variables:

AzureStr: <storage_conneString>

Function app Invocations logs :