How to avoid race condition when updating Azure Table Storage record

Question

Azure Function utilising Azure Table Storage

I have an Azure Function which is triggered from Azure Service Bus topic subscription, let's call it "Process File Info" function.

The message on the subscription contains file information to be processed. Something similar to this:

{
  "uniqueFileId": "adjsdakajksajkskjdasd",
  "fileName":"mydocument.docx",
  "sourceSystemRef":"System1",
  "sizeBytes": 1024,
  ... and other data
}

The function carries out the following two operations -

1. Check individual file storage table for the existing of the file. If it exists, update that file. If it's new, add the file to the storage table (stored on a per system|per fileId basis).

2. Capture metrics on the file size bytes and store in a second storage table, called metrics (constantly incrementing the bytes, stored on a per system|per year/month basis).

The following diagram gives a brief summary of my approach:

The difference between the individualFileInfo table and the fileMetric is that the individual table has one record per file, where as the metric table stores one record per month that is constantly updated (incremented) gathering the total bytes that are passed through the function.

Data in the fileMetrics table is stored as follows:

The issue...

Azure functions are brilliant at scaling, in my setup I have a max of 6 of these functions running at any one time. Presuming each file message getting processed is unique - updating the record (or inserting) in the individualFileInfo table works fine as there are no race conditions.

However, updating the fileMetric table is proving problematic as say all 6 functions fire at once, they all intend to update the metrics table at the one time (constantly incrementing the new file counter or incrementing the existing file counter).

I have tried using the etag for optimistic updates, along with a little bit of recursion to retry should a 412 response come back from the storage update (code sample below). But I can't seem to avoid this race condition. Has anyone any suggestion on how to work around this constraint or come up against something similar before?

Sample code that is executed in the function for storing the fileMetric update:

internal static async Task UpdateMetricEntry(IAzureTableStorageService auditTableService, 
    string sourceSystemReference, long addNewBytes, long addIncrementBytes, int retryDepth = 0)
{
    const int maxRetryDepth = 3; // only recurively attempt max 3 times
    var todayYearMonth = DateTime.Now.ToString("yyyyMM");
    try
    {
        // Attempt to get existing record from table storage.
        var result = await auditTableService.GetRecord<VolumeMetric>("VolumeMetrics", sourceSystemReference, todayYearMonth);

        // If the volume metrics table existing in storage - add or edit the records as required.
        if (result.TableExists)
        {
            VolumeMetric volumeMetric = result.RecordExists ?
                // Existing metric record.
                (VolumeMetric)result.Record.Clone()
                    :
                // Brand new metrics record.
                new VolumeMetric
                {
                    PartitionKey = sourceSystemReference,
                    RowKey = todayYearMonth,
                    SourceSystemReference = sourceSystemReference,
                    BillingMonth = DateTime.Now.Month,
                    BillingYear = DateTime.Now.Year,
                    ETag = "*"
                };

            volumeMetric.NewVolumeBytes += addNewBytes;
            volumeMetric.IncrementalVolumeBytes += addIncrementBytes;

            await auditTableService.InsertOrReplace("VolumeMetrics", volumeMetric);
        }
    }
    catch (StorageException ex)
    {
        if (ex.RequestInformation.HttpStatusCode == 412)
        {
            // Retry to update the volume metrics.
            if (retryDepth < maxRetryDepth)
                await UpdateMetricEntry(auditTableService, sourceSystemReference, addNewBytes, addIncrementBytes, retryDepth++);
        }
        else
            throw;
    }
}

Etag keeps track of conflicts and if this code gets a 412 Http response it will retry, up to a max of 3 times (an attempt to mitigate the issue). My issue here is that I cannot guarantee the updates to table storage across all instances of the function.

Thanks for any tips in advance!!

Answer 1

You can put the second part of the work into a second queue and function, maybe even put a trigger on the file updates.

Since the other operation sounds like it might take most of the time anyways, it could also remove some of the heat from the second step.

You can then solve any remaining race conditions by focusing only on that function. You can use sessions to limit the concurrency effectively. In your case, the system id could be a possible session key. If you use that, you will only have one Azure Function processing data from one system at one time, effectively solving your race conditions.

https://dev.to/azure/ordered-queue-processing-in-azure-functions-4h6c

Edit: If you can't use Sessions to logically lock the resource, you can use locks via blob storage:

https://www.azurefromthetrenches.com/acquiring-locks-on-table-storage/