Polly and Multiple HttpClients

Question

I want to use Polly in conjunction with my HttpClientFactory (in my C# .NET 5.0 project).

But the problem I have is that I have multiple named HttpClients in my factory and what I am trying to achieve is when I have a specific status code e.g. 404 I want to retry by using another named HttpClient from my factory (that points to a different server).

But I see people using the services.AddHttpClient(name, configureOptions).AddPolicyHandler() pattern a lot. But that won't work, because that won't call into another named HttpClient.

Also I want to retry both clients "for them selves" when a 408 happens.

Are there any patterns or example code for this? I don't know what the best way forward is.

Answer 1

I ended up not using the PollyHttpClientBuilderExtensions.AddPolicyHandler but I used this for inspiration to write my own: https://github.com/App-vNext/Polly-Samples/blob/master/PollyDemos/Async/AsyncDemo08_Wrap-Fallback-WaitAndRetry-CircuitBreaker.cs

Answer 2

I think you can achieve the desired the behaviour in the following way.

Let's register two named http client with retry in case of 408 status code:

public void ConfigureServices(IServiceCollection services)
{
    services.AddHttpClient("ServiceA")
        .AddPolicyHandler(GetRetryPolicy("ServiceA"));

    services.AddHttpClient("ServiceB")
        .AddPolicyHandler(GetRetryPolicy("ServiceB"));

    //...
}

private IAsyncPolicy<HttpResponseMessage> GetRetryPolicy(string name)
    => Policy<HttpResponseMessage>
    .HandleResult(res => res.StatusCode == System.Net.HttpStatusCode.RequestTimeout)
    .WaitAndRetryAsync(3, _ => TimeSpan.FromMilliseconds(200),
    onRetryAsync: (dr, ts) => { Console.WriteLine($"Retry by {name}"); return Task.CompletedTask; });

I've provided an onRetryAsync for debugging purposes only to see when does the retry perform.

---

Now let's have a simple webapi controller to wireup things:

[Route("api/[controller]")]
public class HomeController : Controller
{
    private readonly IHttpClientFactory clientFactory;
    public HomeController(IHttpClientFactory clientFactory)
    {
        this.clientFactory = clientFactory;
    }

    [HttpGet]
    public async Task<IActionResult> Get(int expectedStatusCode)
    {
        var result = await GetAsync(GetPrimaryProxy(), expectedStatusCode);
        if(result != null && result.StatusCode == System.Net.HttpStatusCode.NotFound)
            result = await GetAsync(GetSecondaryProxy(), expectedStatusCode);

        return StatusCode(500, result);
    }

    private HttpClient GetPrimaryProxy() => clientFactory.CreateClient("ServiceA");
    private HttpClient GetSecondaryProxy() => clientFactory.CreateClient("ServiceB");

    private async Task<HttpResponseMessage> GetAsync(HttpClient client, int expectedStatusCode)
        => await client.GetAsync($"https://httpstat.us/{expectedStatusCode}");
}

• I've used the https://httpstat.us website to emulate expected response status code
• Unfortunately you have to do the failover manually. The Fallback policy can't be setup in a way that examines HttpResponseMessage and returns an HttpClient.

/api/Home/404

The debug logs:

System.Net.Http.HttpClient.ServiceA.LogicalHandler: Information: Start processing HTTP request GET https://httpstat.us/404
info: System.Net.Http.HttpClient.ServiceA.LogicalHandler[100]
      Start processing HTTP request GET https://httpstat.us/404
info: System.Net.Http.HttpClient.ServiceA.ClientHandler[100]
      Sending HTTP request GET https://httpstat.us/404
System.Net.Http.HttpClient.ServiceA.ClientHandler: Information: Sending HTTP request GET https://httpstat.us/404
info: System.Net.Http.HttpClient.ServiceA.ClientHandler[101]
      Received HTTP response headers after 319.9439ms - 404
System.Net.Http.HttpClient.ServiceA.ClientHandler: Information: Received HTTP response headers after 319.9439ms - 404
info: System.Net.Http.HttpClient.ServiceA.LogicalHandler[101]
      End processing HTTP request after 326.1409ms - 404
System.Net.Http.HttpClient.ServiceA.LogicalHandler: Information: End processing HTTP request after 326.1409ms - 404

info: System.Net.Http.HttpClient.ServiceB.LogicalHandler[100]
      Start processing HTTP request GET https://httpstat.us/404
System.Net.Http.HttpClient.ServiceB.LogicalHandler: Information: Start processing HTTP request GET https://httpstat.us/404
info: System.Net.Http.HttpClient.ServiceB.ClientHandler[100]
      Sending HTTP request GET https://httpstat.us/404
System.Net.Http.HttpClient.ServiceB.ClientHandler: Information: Sending HTTP request GET https://httpstat.us/404
info: System.Net.Http.HttpClient.ServiceB.ClientHandler[101]
      Received HTTP response headers after 344.5937ms - 404
System.Net.Http.HttpClient.ServiceB.ClientHandler: Information: Received HTTP response headers after 344.5937ms - 404
info: System.Net.Http.HttpClient.ServiceB.LogicalHandler[101]
      End processing HTTP request after 350.932ms - 404
System.Net.Http.HttpClient.ServiceB.LogicalHandler: Information: End processing HTTP request after 350.932ms - 404

• ServiceA issued a request and received 404
  • Retry did not trigger since 408 is handled only
• ServiceB issued a request and received 404
  • Retry did not trigger since 408 is handled only

/api/Home/408

The debug logs:

info: System.Net.Http.HttpClient.ServiceA.LogicalHandler[100]
      Start processing HTTP request GET https://httpstat.us/408
System.Net.Http.HttpClient.ServiceA.LogicalHandler: Information: Start processing HTTP request GET https://httpstat.us/408
info: System.Net.Http.HttpClient.ServiceA.ClientHandler[100]
      Sending HTTP request GET https://httpstat.us/408
System.Net.Http.HttpClient.ServiceA.ClientHandler: Information: Sending HTTP request GET https://httpstat.us/408
info: System.Net.Http.HttpClient.ServiceA.ClientHandler[101]
      Received HTTP response headers after 343.5167ms - 408
System.Net.Http.HttpClient.ServiceA.ClientHandler: Information: Received HTTP response headers after 343.5167ms - 408

Retry by ServiceA
info: System.Net.Http.HttpClient.ServiceA.ClientHandler[100]
      Sending HTTP request GET https://httpstat.us/408
System.Net.Http.HttpClient.ServiceA.ClientHandler: Information: Sending HTTP request GET https://httpstat.us/408
info: System.Net.Http.HttpClient.ServiceA.ClientHandler[101]
      Received HTTP response headers after 236.9796ms - 408
System.Net.Http.HttpClient.ServiceA.ClientHandler: Information: Received HTTP response headers after 236.9796ms - 408

Retry by ServiceA
info: System.Net.Http.HttpClient.ServiceA.ClientHandler[100]
      Sending HTTP request GET https://httpstat.us/408
System.Net.Http.HttpClient.ServiceA.ClientHandler: Information: Sending HTTP request GET https://httpstat.us/408
info: System.Net.Http.HttpClient.ServiceA.ClientHandler[101]
      Received HTTP response headers after 207.2602ms - 408
System.Net.Http.HttpClient.ServiceA.ClientHandler: Information: Received HTTP response headers after 207.2602ms - 408

Retry by ServiceA
info: System.Net.Http.HttpClient.ServiceA.ClientHandler[100]
      Sending HTTP request GET https://httpstat.us/408
System.Net.Http.HttpClient.ServiceA.ClientHandler: Information: Sending HTTP request GET https://httpstat.us/408
info: System.Net.Http.HttpClient.ServiceA.ClientHandler[101]
      Received HTTP response headers after 203.3911ms - 408
System.Net.Http.HttpClient.ServiceA.ClientHandler: Information: Received HTTP response headers after 203.3911ms - 408
System.Net.Http.HttpClient.ServiceA.LogicalHandler: Information: End processing HTTP request after 1618.0826ms - 408
info: System.Net.Http.HttpClient.ServiceA.LogicalHandler[101]
      End processing HTTP request after 1618.0826ms - 408

• ServiceA issued a request and received 408
  • Retry is triggered after 200 ms delay
• ServiceA issued a request and received 408
  • Retry is triggered after 200 ms delay
• ServiceA issued a request and received 408
  • Retry is triggered after 200 ms delay
• ServiceA issued a request and received 408
  • Retry is not triggered because max retry count is exceeded
• ServiceB is not called because status code is 408 not 404