Polly Circuit Breaker policy and HttpClient with ASP.NET Core API

Question

I have problems setting up Polly's CircuitBreaker in combination of HttpClient.

Specifically, CircuitBreaker and HttpClient are used for ASP.NET Core Web API Controller following the links below:

https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests

https://github.com/App-vNext/Polly/wiki/Polly-and-HttpClientFactory

Below is what I want

• retry policy: retry 3 times for each request if there are transient error.

• cicuit breaker policy: take effect if fives transient errors occur across all requests.

The problem

Although retry policy works correctly, circuit breaker policy does not work.

The CarController still receives request after 5 exceptions from _httpClient.SendAsync() occurred and doesn't pause for 30 seconds (requests are processed immeidately by the controller).

HandledEventsAllowedBeforeBreaking: 5

DurationOfBreakInSeconds: 30

Am I missing something here?

ConfigureServices

Configure Polly retry and circuit breaker policies, and sepecify custom HttpClient, HttpClientService

    public void ConfigureServices(IServiceCollection services)
        {
           services.AddHttpClient();
           services.AddHttpClient<IHttpClientService, HttpClientService>()                                
                        .AddPolicyHandler((service, request) =>
                            HttpPolicyExtensions.HandleTransientHttpError()
                                .WaitAndRetryAsync(3,
                                    retryCount => TimeSpan.FromSeconds(Math.Pow(2, retryCount)),
                onRetry: (outcome, timespan, retryCount, context) =>
                                {
                                    service.GetService<ILog>().Error("Delaying for {delay}ms, then making retry {retry}.",
                                        timespan.TotalMilliseconds, retryCount);
                                }
                            )
)
                        )
                       .AddPolicyHandler((service, request) =>                      
                                HttpPolicyExtensions.HandleTransientHttpError()
                                    //Further external requests are blocked for 30 seconds if five failed attempts occur sequentially.
                                    //Circuit breaker policies are stateful.All calls through this client share the same circuit state.
                                    .CircuitBreakerAsync(5,
                                                 TimeSpan.FromSeconds(30), 
                                                 (result, timeSpan, context)=>
                                                            service.GetService<ILog>().Error("CircuitBreaker onBreak for {delay}ms", timeSpan.TotalMilliseconds),
                                                  context =>
                                                      service.GetService<ILog>().Error("CircuitBreaker onReset")));

         }

CarController

IHttpClientService is specified in Polly policy in ConfigureServices. HttpClientService uses HttpClient.

Circuit breaker does not work: even after five transient error occurs (e.g. HttpRequestException) from _httpClient.SendAsync(), CarController can still receive request, and doesn't pause for 30 seconds.

 [ApiVersion("1")]
    [Route("api/v{version:apiVersion}/[controller]")]
    [ApiController]
    public class CarController : ControllerBase
    {
        private readonly ILog _logger;
        private readonly IHttpClientService _httpClientService;
        private readonly IOptions<Config> _config;

        public CarController(ILog logger, IHttpClientService httpClientService, IOptions<Config> config)
        {
            _logger = logger;
            _httpClientService = httpClientService;
            _config = config;
        }

        [HttpPost]
        public async Task<ActionResult> Post()
        {  

            using (StreamReader reader = new StreamReader(Request.Body, Encoding.UTF8))
            {
                string body = reader.ReadToEnd();

                    var statusCode = await _httpClientService.PostAsync(
                        "url",
                        new Dictionary<string, string>
                        {
                            {"headerID", "Id"}                           
                        },
                        body);
                    return StatusCode((int)statusCode);               
            }
        }
      }

HttpClientService

It seems HttpClient is not stateful across requests.

Circuit breaker does not work: even after five transient error occurs (e.g. HttpRequestException) from _httpClient.SendAsync(), CarController can still receive request, and doesn't pause for 30 seconds.

 public class HttpClientService
{
    private readonly HttpClient _httpClient;
    public HttpClientService(HttpClient client)
    {
        _httpClient = client;
    }

    public async Task<HttpStatusCode> PostAsync(string url, Dictionary<string, string> headers, string body)
    {
        using (var content = new StringContent(body, Encoding.UTF8, "application/json"))
        {
            foreach (var keyValue in headers)
            {
                content.Headers.Add(keyValue.Key, keyValue.Value);
            }

             var request = new HttpRequestMessage(HttpMethod.Post, url)
                                    {
                                        Content = content
                                    };

            var response = await _httpClient.SendAsync(request);

            response.EnsureSuccessStatusCode();
            return response.StatusCode;
        }

    }

ASP.NET Core API 2.2

Update Updated SetWaitAndRetryPolicy extension method to use IServiceProvider.

Answer 1

Circuit-breaker policies are stateful to track failure rates across calls, and so need to be long-lived rather than created per request.

The way the overload on HttpClientFactory in the code posted is used:

.AddPolicyHandler((service, request) => HttpPolicyExtensions.HandleTransientHttpError()
    .CircuitBreakerAsync( /* etc */

is manufacturing an instance of the circuit-breaker per request, so the circuit-breakers never have time to build up a failure state.

That overload (and similars) are designed for selecting policies dynamically based on characteristics of the request. But in fact the posted code is (EDIT: the originally posted code was) not making any use of the service, request input parameters, so you can just delete the (service, request) => part, and make use of an overload on HttpClientFactory which takes a policy instance:

.AddPolicyHandler(HttpPolicyExtensions.HandleTransientHttpError()
    .WaitAndRetryAsync(/* etc */))
.AddPolicyHandler(HttpPolicyExtensions.HandleTransientHttpError()
    .CircuitBreakerAsync(/* etc */))

That single, long-lived instance of circuit-breaker returned by HttpPolicyExtensions.HandleTransientHttpError().CircuitBreakerAsync(/* etc */) will then be used by the instances of HttpClient for which HttpClientFactory was configured to use it.

---

Sidenote (especially for any readers who do want to use circuit-breaker with the given overloads):

It is possible to use circuit-breaker with the request-driven policySelector overloads in HttpClientFactory. It is just necessary to ensure that a single instance is selected by the lambda expression, not that a new instance is manufactured each time per request. For example:

var circuitBreaker = HttpPolicyExtensions.HandleTransientHttpError().CircuitBreakerAsync(/* etc */);
services.AddHttpClient<IHttpClientService, HttpClientService>()                                
    .AddPolicyHandler((service, request) => circuitBreaker); // By way of example technique: more typically with this overload, there is some more complex logic to select different policies for different kinds of request.

EDIT to answer question in comments: That instance does not have to be declared static to make it long-lived. It can be declared within the Startup.ConfigureServices(...) method, immediately before use, as per the code example above. The lambda and configuring it on HttpClientFactory will capture it and make it long-lived.

The circuitBreaker instance should be shared across calls you want to break in common. If you attach a circuit-breaker to a particular HttpClient configuration declared via HttpClientFactory, all calls through instances of that HttpClient configuration later retrieved from HttpClientFactory by DI, will share the circuitBreaker, and thus break in common.

When using circuit-breaker with HttpClientFactory, this typically means you might declare one HttpClient configuration (typed or named) on HttpClientFactory per subsystem for which you want calls to circuit-break in common.

---

Sidenote: The variant of circuit-breaker selected also triggers based on consecutive fault count. (Mentioned just in case an extra factor; the question posted refers to 5 errors occurring across requests but not specifically consecutively.)