Identifying which Polly circuit breaker tripped in the OnOpened callback

Question

I have several named HttpClients in my C# application. I am binding Polly policies (namely the circuit breaker) to each of them.

I need to capture which HttpClient tripped the circuit in the OnOpened callback so I can report back to the caller that ClientX is down.

I see there is a OperationKey property on the OnCircuitOpenedArguments, and I think this is what I want, but I can't figure out how to populate it.

Note I am using Polly v8 and Microsoft.Extensions.Http.Resilience.

Here's the code I have so far:

In my DI setup

{
  services
    .AddHttpClient("ClientA")
    .AddResilienceHandler("ClientA", (builder, context) =>
    {
      var strategyFactory = context.ServiceProvider.GetRequiredService<IResiliencyStrategyFactory>();
      builder.AddPipeline(strategyFactory.Create());
    });

  services
    .AddHttpClient("ClientB")
    .AddResilienceHandler("ClientB", (builder, context) =>
    {
      var strategyFactory = context.ServiceProvider.GetRequiredService<IResiliencyStrategyFactory>();
      builder.AddPipeline(strategyFactory.Create());
    });
}

The factory method called above

public ResiliencePipeline<HttpResponseMessage> Create()
{
  var builder = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddTimeout(MapToTimeoutStrategyOptions())
    .AddCircuitBreaker(MapToHttpCircuitBreakerStrategyOptions())
    .AddRetry(MapToServiceNowHttpRetryStrategyOptions());
}

The circuit breaker definition

private HttpCircuitBreakerStrategyOptions MapToHttpCircuitBreakerStrategyOptions() =>
  new()
  {
    MinimumThroughput = resiliencyConfig.CircuitBreakerConfig.Throughput,
    FailureRatio = resiliencyConfig.CircuitBreakerConfig.FailureRatio,
    SamplingDuration = TimeSpan.FromSeconds(resiliencyConfig.CircuitBreakerConfig.SamplingDuration),
    BreakDuration = TimeSpan.FromSeconds(resiliencyConfig.CircuitBreakerConfig.DurationOfBreakSeconds),
    OnClosed = args =>
    {
      //Signal that ClientX is back up
    },
    OnOpened = args =>
    {
      //Signal that ClientX is down
    }
  };

Answer 1

There are multiple ways how can you get notified when one of CB breaks or transitions back to closed state.

Using closure

You can simply pass the httpclient's name through the method call chain and then you can use that inside the OnXYZ callback methods via closure.

DI

{
  services
    .AddHttpClient("ClientA")
    .AddResilienceHandler("ClientA", (builder, context) =>
    {
      var strategyFactory = context.ServiceProvider.GetRequiredService<IResiliencyStrategyFactory>();
      builder.AddPipeline(strategyFactory.Create("ClientA"));
    });

  services
    .AddHttpClient("ClientB")
    .AddResilienceHandler("ClientB", (builder, context) =>
    {
      var strategyFactory = context.ServiceProvider.GetRequiredService<IResiliencyStrategyFactory>();
      builder.AddPipeline(strategyFactory.Create("ClientB"));
    });
}

Factory method

public ResiliencePipeline<HttpResponseMessage> Create(string httpClientName)
{
  var builder = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddTimeout(MapToTimeoutStrategyOptions())
    .AddCircuitBreaker(MapToHttpCircuitBreakerStrategyOptions(httpClientName))
    .AddRetry(MapToServiceNowHttpRetryStrategyOptions());
}

Strategy definition

private HttpCircuitBreakerStrategyOptions MapToHttpCircuitBreakerStrategyOptions(string httpClientName) =>
  new()
  {
    Name = $"CircuitBreakerFor{httpClientName}",
    MinimumThroughput = resiliencyConfig.CircuitBreakerConfig.Throughput,
    FailureRatio = resiliencyConfig.CircuitBreakerConfig.FailureRatio,
    SamplingDuration = TimeSpan.FromSeconds(resiliencyConfig.CircuitBreakerConfig.SamplingDuration),
    BreakDuration = TimeSpan.FromSeconds(resiliencyConfig.CircuitBreakerConfig.DurationOfBreakSeconds),
    OnClosed = args =>
    {
      //receive/retrieve logger
      logger.LogInformation("CB for {HttpClient} has closed.", httpClientName);
      return default; 
    },
    OnOpened = args =>
    {
      //receive/retrieve logger
      logger.LogError("CB for {HttpClient} has opened.", httpClientName);
      return default; 
    }
  };

Telemetry

If you just want to log the fact that the CB has transitioned into a new state then you can rely on the built-in Telemetry support.

You can easily set up any LoggerFactory via the TelemetryOptions.

Here are some sample telemetry logs:

Resilience event occurred. EventName: 'OnCircuitOpened', Source: 'MyPipeline/MyPipelineInstance/MyCircuitBreakerStrategy', Operation Key: 'MyCircuitedOperation', Result: 'Exception of type 'CustomException' was thrown.'
    CustomException: Exception of type 'CustomException' was thrown.
        at Program.<>c.<<Main>b__0_1>d.MoveNext()
        ...
        at Polly.ResiliencePipeline.<>c__8`1.<<ExecuteAsync>b__8_0>d.MoveNext() in /_/src/Polly.Core/ResiliencePipeline.AsyncT.cs:line 95

Resilience event occurred. EventName: 'OnCircuitHalfOpened', Source: 'MyPipeline/MyPipelineInstance/MyCircuitBreakerStrategy', Operation Key: 'MyCircuitedOperation', Result: ''

Resilience event occurred. EventName: 'OnCircuitClosed', Source: 'MyPipeline/MyPipelineInstance/MyCircuitBreakerStrategy', Operation Key: 'MyCircuitedOperation', Result: '42'

In order to populate the Source part with http client aware names you should set the CircuitBreakerStrategyOptions{<Result>}'s Name property. Just like we did under the strategy definition section above.

private HttpCircuitBreakerStrategyOptions MapToHttpCircuitBreakerStrategyOptions(string httpClientName) =>
  new()
  {
    Name = $"CircuitBreakerFor{httpClientName}",
    MinimumThroughput = resiliencyConfig.CircuitBreakerConfig.Throughput,
    FailureRatio = resiliencyConfig.CircuitBreakerConfig.FailureRatio,
    SamplingDuration = TimeSpan.FromSeconds(resiliencyConfig.CircuitBreakerConfig.SamplingDuration),
    BreakDuration = TimeSpan.FromSeconds(resiliencyConfig.CircuitBreakerConfig.DurationOfBreakSeconds)
  };