Can we use Polly retry instead of ExponentialBackoffRetry in Service Bus Topic Trigger Azure Function?

Question

We are using Service Bus Topic Trigger Azure function, and we’re planning to implement a simple behavior in Azure Function, if there is any exception during processing/handling we want to postpone the next retry from some time.

Currently we're planning to use [ExponentialBackoffRetry] attribute, as shown in the below code.

But Can we use Polly retry instead of [ExponentialBackoffRetry]? Basically which approach would be idle for our requirement - [ExponentialBackoffRetry] or Polly retry

Below is our Service Bus Topic Trigger Azure Function:

 [FunctionName(nameof(CardGroupEventSubscriber))]
 [ExponentialBackoffRetry(5, "00:00:04", "00:01:00")]
 public async Task RunAsync([ServiceBusTrigger("%ServiceBusConfigOptions:TopicEventTypeName%", "%ServiceBusConfigOptions:TopicEventTypeSubscription%",
            Connection = "ServiceBusConfigOptions:ConnectionString")]
            string sbMsg)
        {
            try
            {   
                var message = sbMsg.AsPoco<CardGroupEvent>();

                _logger.LogInformation("{class} - {method} - {RequestId} - Start",
                   nameof(CardGroupEventSubscriber), nameof(CardGroupEventSubscriber.RunAsync), message.RequestID);

                _logger.LogInformation($"Started processing message {message.AsJson()} with", nameof(CardGroupEventSubscriber));

                var validationResult = new CardGroupEventValidator().Validate(message);

                if (validationResult.IsValid)
                {
                    await _processor.ProcessAsync(message);
                }
                
            catch (Exception ex)
            {
                _logger.LogError($"Unable to process card group event {sbMsg.AsJson()} with {nameof(CardGroupEventSubscriber)}," +
                    $" ExceptionMessage:{ex.Message}, StackTrace: {ex.StackTrace}");
                throw;
            }
            #endregion

        }

Answer 1

Polly's policies can be defined and used in an imperative way.
Whereas the ExponentialBackoffRetry attribute can be considered as declarative.

So, let's say you want to define a policy which

• Do an exponential back-off with jitter
• Only if a CosmosException is thrown then you do that like this:

const int maxRetryAttempts = 10;
const int oneSecondInMilliSeconds = 1000;
const int maxDelayInMilliseconds = 32 * oneSecondInMilliSeconds;
var jitterer = new Random();
var policy = Policy
  .Handle<CosmosException>()
  .WaitAndRetryAsync(
      maxRetryAttempts,
      retryAttempt =>
      {
          var calculatedDelayInMilliseconds = Math.Pow(2, retryAttempt) * oneSecondInMilliSeconds;
          var jitterInMilliseconds = jitterer.Next(0, oneSecondInMilliSeconds);

          var actualDelay = Math.Min(calculatedDelayInMilliseconds + jitterInMilliseconds, maxDelayInMilliseconds);
          return TimeSpan.FromMilliseconds(actualDelay);
      }
  );

• I have implemented here by my own the exponential back-off
  • The main reason is to reduce the number of packages (so, we don't need the Polly.Contrib.WaitAndRetry)

Now let's apply this to your RunAsync method

[FunctionName(nameof(CardGroupEventSubscriber))]
public async Task RunAsync([ServiceBusTrigger("%ServiceBusConfigOptions:TopicEventTypeName%", "%ServiceBusConfigOptions:TopicEventTypeSubscription%",
    Connection = "ServiceBusConfigOptions:ConnectionString")]
    string sbMsg)
  => await GetExponentialBackoffRetryPolicy.ExecuteAsync(() => RunCoreAsync(sbMsg));

private async Task RunCoreAsync(string sbMsg)
{
    try
    ...
}

• I have moved the original RunAsync's code into the RunCoreAsync method
• I have replaced the RunAsync implementation with a one liner which creates the above policy then decorates the RunCoreAsync

---

Just a side note: In case of CosmosDb it might make sense to handle the rate limiting/throttling in a different way.

When I receive a CosmosException and the StatusCode is 429 then use the RetryAfter's Value to delay the retry, something like this

var policy = Policy
    .Handle<CosmosException>(ex => ex.StatusCode == HttpStatusCode.TooManyRequests)
    .WaitAndRetryAsync(maxRetryAttempts,
       sleepDurationProvider:(_, ex, __) => ((CosmosException)ex).RetryAfter.Value,
       onRetryAsync: (_, __, ___, ____) => Task.CompletedTask);

---

UPDATE #1: Combining the two policies

If you want you can combine the above two policies. All you need to do is to make them independent. So, whatever happens only one of the policies should be triggered. The easiest solution is to pass this ex => ex.StatusCode != HttpStatusCode.TooManyRequests predicate to the exponential backoff policy

IAsyncPolicy GetExponentialBackoffRetryPolicy()
    => Policy
    .Handle<CosmosException>(ex => ex.StatusCode != HttpStatusCode.TooManyRequests)
    .WaitAndRetryAsync(
        maxRetryAttempts,
        retryAttempt =>
        {
            var calculatedDelayInMilliseconds = Math.Pow(2, retryAttempt) * oneSecondInMilliSeconds;
            var jitterInMilliseconds = jitterer.Next(0, oneSecondInMilliSeconds);

            var actualDelay = Math.Min(calculatedDelayInMilliseconds + jitterInMilliseconds, maxDelayInMilliseconds);
            return TimeSpan.FromMilliseconds(actualDelay);
        }
    );

IAsyncPolicy GetThrottlingAwareRetryPolicy()
    => Policy
    .Handle<CosmosException>(ex => ex.StatusCode == HttpStatusCode.TooManyRequests)
    .WaitAndRetryAsync(maxRetryAttempts,
        sleepDurationProvider: (_, ex, __) => ((CosmosException)ex).RetryAfter.Value,
        onRetryAsync: (_, __, ___, ____) => Task.CompletedTask);

In order to combine these two into one you have many options, I suggest to use the Policy.WrapAsync

IAsyncPolicy retryPolicy = Policy.WrapAsync(GetExponentialBackoffRetryPolicy(), GetThrottlingAwareRetryPolicy());
//OR
IAsyncPolicy retryPolicy = Policy.WrapAsync(GetThrottlingAwareRetryPolicy(), GetExponentialBackoffRetryPolicy());

Here the ordering does not matter since they are independent policies.