Proper way to handle multiple services with polly circuit breaker

Question

I have an application where we communicate with hundreds of HTTPs endpoints. The application is a proxy of sorts.

When testing with polly, I've noticed that if one endpoint, say api.endpoint1.com fails, the calls to api.endpoint2.com and api.endpoint3.com will also be in an open/blocked state.

This makes sense as I've only defined one policy, but what is the recommended approach to handling this scenario so that calls to unrelated endpoints are not blocked due to another having performance issues?

Do I create a collection of Policy's, one for each endpoint or is there a way to supply a context key of sorts(i.e. the hostname) to scope the failures to a given host endpoint?

I've reviewed Polly's docs regarding context keys and it appears these are a way to exchange data back and forth and not what I'm looking for here.

var policy = Policy
            .Handle<TimeoutException>()
            .CircuitBreaker(1, TimeSpan.FromSeconds(1));
                
//dynamic, large list of endpoints. 
var m = new HttpRequestMessage(HttpMethod.Post, "https://api.endpoint1.com")
{
    Content = new StringContent("some JSON data here", Encoding.UTF8,"application/json")
};


policy.Execute(() => HTTPClientWrapper.PostAsync(message));

Answer 1

Here is my alternative solution which does not maintain a collection of policies (either via an IDictionary or via an IConcurrentPolicyRegistry) rather it takes advantage of named typed clients. (Yes you have read correctly named and typed HttpClients)

The named and typed clients

Most probably you have heard (or even used) named or typed clients. But I'm certain that you haven't used named and typed clients. It is a less documented feature of HttpClientFactory + HttpClient combo.

If you look at the different overloads of the AddHttpClient extension method then you can spot this one:

public static IHttpClientBuilder AddHttpClient<TClient,TImplementation> 
   (this IServiceCollection services, string name, Action<HttpClient> configureClient) 
   where TClient : class where TImplementation : class, TClient;

It allows us to register a typed client and give a logical name to it. But how can I get the proper instance? That's where the ITypedHttpClientFactory comes into the picture. It allows us to create a typed client from a named client. Wait what??? I hope you will understand this sentence at the end of this post. :)

The typed client

For the sake of simplicity let me use this typed client as an example:

public interface IResilientClient
{
    Task GetAsync();
}

public class ResilientClient: IResilientClient
{
    private readonly HttpClient client;

    public ResilientClient(HttpClient client)
    {
        this.client = client;
    }

    public Task GetAsync()
    {
        //TODO: implement it properly
        return Task.CompletedTask;
    }
}

The named and typed clients registration

Let suppose you have a list of downstream system urls (urls). Then you can register multiple typed client instances with different unique names and base urls

foreach (string url in urls)
{
    builder.Services
      .AddHttpClient<IResilientClient, ResilientClient>(url,
          client => client.BaseAddress = new Uri(url))
      .AddPolicyHandler(GetCircuitBreakerPolicy());
}

• Here I have used the url as the unique name
  • So, we can get the appropriate instance based on the downstream url

The policy definition

private IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
   => Policy<HttpResponseMessage>
            .Handle<TimeoutException>()
            .CircuitBreakerAsync(1, TimeSpan.FromSeconds(1));

• I have modified the policy to support async: .CircuitBreakerAsync
• I've also amended it to be suitable with the AddPolicyHandler: Policy<HttpResponseMessage>
• It is defined as a function so each registered named typed client will have a different Circuit Breaker instance

The usage

This is be a bit clumsy, but I think it is okay. So, wherever you want to use one of the named typed clients you have to inject two interfaces:

• IHttpClientFactory: To be able to create a named HttpClient
• ITypedHttpClientFactory<ResilientClient>: To be able to create a typed client from the named HttpClient

public XYZService(
   IHttpClientFactory namedClientFactory, 
   ITypedHttpClientFactory<ResilientClient> namedTypedClientFactory)
{
    var namedClient = namedClientFactory.CreateClient(xyzUrl);
    var namedTypedClient = namedTypedClientFactory.CreateClient(namedClient);
}

• Please note that you have to use ResilientClient concrete class as the type parameter not the interface IResilientClient
  • If you would use the interface then you would receive the following runtime error:

InvalidOperationException: A suitable constructor for type 'IResilientClient' could not be located. Ensure the type is concrete and all parameters of a public constructor are either registered as services or passed as arguments. Also ensure no extraneous arguments are provided.

Summary

• With the named and typed client feature of AddHttpClient we can register multiple instances of the same typed client
• With the IHttpClientFactory we can retrieve a registered named client which has the proper BaseAddress and decorated with a Circuit Breaker
• With the ITypedHttpClientFactory we can convert the named client into a typed client to be able to hide low-level API usage

---

Related sample application's github repository

Answer 2

Yes, your best bet is to create a separate policy per endpoint. This is better than doing it per host because an endpoint may be slow responding for a reason that's specific to that endpoint (e.g., stored procedure is slow).

I've used a Dictionary<string, Policy> with the endpoint URL as the key.

if (!_circuitBreakerPolices.ContainsKey(url))
{
    CircuitBreakerPolicy policy = Policy.Handle<Exception>().AdvancedCircuitBreakerAsync(
        onBreak: ...
    );
    _circuitBreakerPolicies.Add(url, policy);
}
await _circuitBreakerPolicies[url].ExecuteAsync(async () => ... );