CODEWITHSUNDEEP
azure-functionsazure-blob-storage
PSXYU
PSXYU

Reputation: 111

Azure Functions: Error: 409 LeaseIdMismatchWithLeaseOperation (Listener)

We are using Azure Functions (V2 – Consumption Plan). Occasionally (Perhaps once or twice a day) we get a series of the below error.

I have tried Googling it to figure out how to solve it – but for the time being with no success.

Any idea how I could solve it? Are there errors something I should be concerned about – or is it normal to receive them?

Singleton lock renewal failed for blob 'xxxxxxx/xxxxx.xxxx.RunAsync.Listener' with error code 409: LeaseIdMismatchWithLeaseOperation. The last successful renewal completed at 2020-07-24T05:00:16.732Z (33347 milliseconds ago) with a duration of 4 milliseconds. The lease period was 15000 milliseconds.

Accompanied with this error

Exception: Microsoft.Azure.Storage.StorageException: The lease ID specified did not match the lease ID for the blob.
at Microsoft.Azure.Storage.Core.Executor.Executor.ExecuteAsync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
at Microsoft.Azure.WebJobs.Host.StorageBaseDistributedLockManager.SingletonLockHandle.RenewAsync(ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host.Storage\Singleton\BlobLeaseDistributedLockManager.cs:line 348
at Microsoft.Azure.WebJobs.Host.SingletonManager.RenewLeaseCommand.ExecuteAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonManager.cs:line 332
at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.RunAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\TaskSeriesTimer.cs:line 147
Request Information
RequestID:708f6f2f-d01e-013a-1677-xxxxxxx
RequestDate:Fri, 24 Jul 2020 05:00:48 GMT
StatusMessage:The lease ID specified did not match the lease ID for the blob.
ErrorCode:LeaseIdMismatchWithLeaseOperation
ErrorMessage:The lease ID specified did not match the lease ID for the blob. RequestId:708f6f2f-d01e-013a-1677-xxxxx Time:2020-07-24T05:00:48.4767302Z

Upvotes: 2

Views: 10493

Answers (3)

aprobinda
aprobinda

Reputation: 21

In my case, the error occurred when a service class tried to release the blob lease after the lease period has expired because another AcquireBlobLeaseAsync request succeeded:

The lease ID specified did not match the lease ID for the blob.
Status: 409 (The lease ID specified did not match the lease ID for the blob.)
ErrorCode: LeaseIdMismatchWithLeaseOperation

The service class is a wrapper for BlobLeaseClient class from namespace Azure.Storage.Blobs.Specialized

Here's how I reproduced the error with an integration test:

[TestMethod]
public async Task BlobLeaseService_ReleaseBlobLease_WhenLeaseExpired_AndNewLeaseWasAcquiredForTheSamePartitionKey()
{
  var firstJob = CreateJob();
  var secondJob = CreateJob();
  IBlobLeaseService sut = new BlobLeaseService(_connectionString, BlobLeaseDuration.Minimum);

  string firstLeaseId = await sut.AcquireBlobLeaseAsync(firstJob.PartitionKey, _cancellationToken);
  await Task.Delay(TimeSpan.FromSeconds(3 + (int)BlobLeaseDuration.Minimum));//Let the lease period expire while the job is running
  string secondLeaseId = await sut.AcquireBlobLeaseAsync(secondJob.PartitionKey, _cancellationToken);
  await Task.Delay(TimeSpan.FromSeconds(3));
  await sut.ReleaseBlobLeaseAsync(secondJob.PartitionKey, secondLeaseId, _cancellationToken);

  await sut.ReleaseBlobLeaseAsync(firstJob.PartitionKey, firstLeaseId, _cancellationToken);
}

I fixed the error with a catch block:

public async Task ReleaseBlobLeaseAsync(string partitionKey, string leaseId, CancellationToken cancellationToken)
{
  if (cancellationToken.IsCancellationRequested) return;
  BlobLeaseClient blobLeaseClient = GetBlobLeaseClient(partitionKey, leaseId);
  try
  {
      await blobLeaseClient.ReleaseAsync(cancellationToken: cancellationToken);
  }
  catch (Azure.RequestFailedException ex) when (ex.ErrorCode == "LeaseIdMismatchWithLeaseOperation")
  {/*ignore: the leaseId's lease period has expired because another AcquireBlobLeaseAsync request succeeded. */}
}

Upvotes: 1

T. Alpers
T. Alpers

Reputation: 66

In my case, some logic running in a different thread threw an exception. So I removed the multithreading and the error messages are readable again.

If you run Azure Functions in Kubernetes, there is a fuctionsTimeout variable in the host.json, you can set it to -1, that means no timeout. It's not officially documented for Kubernetes purposes, but it works for me.

Also, be sure to use the [Singleton] annotation so that the function is not executed in parallel

Upvotes: 1

Frank Borzage
Frank Borzage

Reputation: 6796

This seems to be a problem that has existed for a long time. I checked some similar problems. Although I did not find a solution, this answer seems to have solved some of my confusion:

The cause is almost always due to functions blocking threads that prevent our lease renewal timers to fire, thus the leases expire unexpectedly. My recommendation for that issue is to evaluate your function code for potential threading issues -- make sure it's using async everywhere possible and not blocking any threads.

Upvotes: 3

Related Questions