How to handle append blob exception?

Question

How do I properly handle the “"The committed block count cannot exceed the maximum limit of 50,000 blocks." Exception when doing a CloudAppendBlob.AppendTextAsync?

I currently have the following code:

    private async Task MessageLogger()
    {
        string messages = "";
        while (true)
        {
            // Check if a new log file should be created
            if (DateTime.UtcNow.Day != _logCreatedUtc.Day)
            {
                // Create a new log file
                await CreateNewLogAsync();
            }

            if (_messageQueue.Count == 0 && messages.Length == 0)
            {
                await Task.Delay(50);
                continue;
            }

            int n = 0;
            while ((this._messageQueue.Count > 0) && (n < 50))
            {
                string message;
                if (this._messageQueue.TryPeek(out message))
                {
                    messages += message;
                    this._messageQueue.TryDequeue(out message);
                    n++;
                }
            }

            try
            {
                // Append messages to Azure Blob
                await _appendBlob.AppendTextAsync(messages);
                messages = "";
            }
            catch (Microsoft.WindowsAzure.Storage.StorageException exception)
                when(exception.RequestInformation.HttpStatusCode == 404)
            {
                // Log file was deleted. Create a new log file
                await CreateNewLogAsync();
            }
            catch (Exception exception)
            {
                LogException("Exception from CloudAppendBlob.AppendTextAsync", exception);
                await Task.Delay(1000);
            }
        }
    }

From time to time, I get the following kind of exception logged as a result of the generic exception handler in my code:

01.04.2018 22:17:03.030 - <<EXCEPTION>>: Exception from CloudAppendBlob.AppendTextAsync
01.04.2018 22:17:03.044 - <<EXCEPTION>>: Type =<System.AggregateException> Message =<One or more errors occurred. (One or more errors occurred. (The committed block count cannot exceed the maximum limit of 50,000 blocks.))> InnerException.Message=<One or more errors occurred. (The committed block count cannot exceed the maximum limit of 50,000 blocks.)>
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.<Dispose>b__8_0()
   at Microsoft.WindowsAzure.Storage.Core.Util.CommonUtility.RunWithoutSynchronizationContext(Action actionToRun)
   at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at Microsoft.WindowsAzure.Storage.Blob.CloudAppendBlob.<UploadFromStreamAsyncHelper>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Infrastructure.Logging.AzureBlob.Logger.<MessageLogger>d__24.MoveNext() in /opt/vsts/work/1/s/Infrastructure/Logging/Infrastructure.Logging.AzureBlob/Logger.cs:line 183

How does an exception handler look like to catch exactly this kind of error? I would like to add a specific handler for it.

I have tried to look in the debugger, but it looks very strange:

The exception variable looks like it's "null" in the debugger, but in reality it's not and it gets logged ok.

Answer 1

Your exception is wrapped in an AggregateException. This is how you can unwrap and handle it:

catch (Exception exception)
{
    ExtractStorageError(exception);
}

. . .

private static string ExtractStorageError(Exception ex) {   
    while (ex != null && !(ex is StorageException))     
    {       
        ex = (ex.InnerException != null) ? ex.InnerException.GetBaseException() : ex.InnerException;
    }   
    if (ex is StorageException)     
    {   
        StorageException se = ex as StorageException;    
        if(ste.RequestInformation.ExtendedErrorInformation != null)
           return se.RequestInformation.ExtendedErrorInformation.ErrorCode;     
    }   
    return null; 
}

Answer 2

I checked with the product team, and they advised to implement and use AccessConditions for block append to avoid the exception from happening:

 /// <summary>
    /// Gets or sets a value for a condition that specifies the maximum size allowed for an append blob when a new block is committed. The append
    /// will succeed only if the size of the blob after the append operation is less than or equal to the specified size.
    /// </summary>
    /// <value>The maximum size in bytes, or <c>null</c> if no value is set.</value>
    /// <remarks>This condition only applies to append blobs.</remarks>
    public long? IfMaxSizeLessThanOrEqual

Essentially you will be checking the number of blocks, and based on the condition, you could start a new blob to append to(if the max is reached), or continue appending.

These two would be helpful:

x-ms-blob-condition-maxsize: Clients can use this header to ensure that append operations do not increase the blob size beyond an expected maximum size in bytes. If the condition fails, the request will fail with MaxBlobSizeConditionNotMet error (HTTP status code 412 – Precondition Failed).

x-ms-blob-condition-appendpos Optional conditional header, used only for the Append Block operation. A number indicating the byte offset to compare. Append Blockwill succeed only if the append position is equal to this number. If it is not, the request will fail with the AppendPositionConditionNotMet error (HTTP status code 412 – Precondition Failed).

Source:** https://github.com/Azure/azure-storage-net/blob/master/Lib/Common/AccessCondition.cs**

https://learn.microsoft.com/en-us/rest/api/storageservices/append-block