Spring Cloud - Getting Retry Working In RestTemplate?

Question

I have been migrating an existing application over to Spring Cloud's service discovery, Ribbon load balancing, and circuit breakers. The application already makes extensive use of the RestTemplate and I have been able to successfully use the load balanced version of the template. However, I have been testing the situation where there are two instances of a service and I drop one of those instances out of operation. I would like the RestTemplate to failover to the next server. From the research I have done, it appears that the fail-over logic exists in the Feign client and when using Zuul. It appears that the LoadBalancedRest template does not have logic for fail-over. In diving into the code, it looks like the RibbonClientHttpRequestFactory is using the netflix RestClient (which appears to have logic for doing retries).

So where do I go from here to get this working?

I would prefer to not use the Feign client because I would have to sweep A LOT of code. I had found this link that suggested using the @Retryable annotation along with @HystrixCommand but this seems like something that should be a part of the load balanced rest template.

I did some digging into the code for RibbonClientHttpRequestFactory.RibbonHttpRequest:

protected ClientHttpResponse executeInternal(HttpHeaders headers) throws IOException {

    try {
        addHeaders(headers);
        if (outputStream != null) {
            outputStream.close();
            builder.entity(outputStream.toByteArray());
        }

        HttpRequest request = builder.build();
        HttpResponse response = client.execute(request, config);

        return new RibbonHttpResponse(response);
    }
    catch (Exception e) {
        throw new IOException(e);
    }
}

It appears that if I override this method and change it to use "client.executeWithLoadBalancer()" that I might be able to leverage the retry logic that is built into the RestClient? I guess I could create my own version of the RibbonClientHttpRequestFactory to do this?

Just looking for guidance on the best approach.

Thanks

Answer 1

I had the same problem but then, out of the box, everything was working (using a @LoadBalanced RestTemplate). I am using Finchley version of Spring Cloud, and I think my problem was that I was not explicity adding spring-retry in my pom configuration. I'll leave here my spring-retry related yml configuration (remember this only works with @LoadBalanced RestTemplate, Zuul of Feign):

spring:
# Ribbon retries on
  cloud:
    loadbalancer:
      retry:
        enabled: true

# Ribbon service config
my-service:
  ribbon:
    MaxAutoRetries: 3
    MaxAutoRetriesNextServer: 1
    OkToRetryOnAllOperations: true
    retryableStatusCodes: 500, 502

Answer 2

To answer my own question:

Before I get into the details, a cautionary tale:

Eureka's self preservation mode sent me down a rabbit hole while testing the fail-over on my local machine. I recommend turning self preservation mode off while doing your testing. Because I was dropping nodes at a regular rate and then restarting (with a different instance ID using a random value), I tripped Eureka's self preservation mode. I ended up with many instances in Eureka that pointed to the same machine, same port. The fail-over was actually working but the next node that was chosen happened to be another dead instance. Very confusing at first!

I was able to get fail-over working with a modified version of RibbonClientHttpRequestFactory. Because RibbonAutoConfiguration creates a load balanced RestTemplate with this factory, rather then injecting this rest template, I create a new one with my modified version of the request factory:

protected RestTemplate restTemplate;

@Autowired
public void customizeRestTemplate(SpringClientFactory springClientFactory, LoadBalancerClient loadBalancerClient) {
    restTemplate = new RestTemplate();

    // Use a modified version of the http request factory that leverages the load balacing in netflix's RestClient.
    RibbonRetryHttpRequestFactory lFactory = new RibbonRetryHttpRequestFactory(springClientFactory, loadBalancerClient);
    restTemplate.setRequestFactory(lFactory);
}

The modified Request Factory is just a copy of RibbonClientHttpRequestFactory with two minor changes:

1) In createRequest, I removed the code that was selecting a server from the load balancer because the RestClient will do that for us. 2) In the inner class, RibbonHttpRequest, I changed executeInternal to call "executeWithLoadBalancer".

The full class:

@SuppressWarnings("deprecation")
public class RibbonRetryHttpRequestFactory implements ClientHttpRequestFactory {

    private final SpringClientFactory clientFactory;
    private LoadBalancerClient loadBalancer;

    public RibbonRetryHttpRequestFactory(SpringClientFactory clientFactory, LoadBalancerClient loadBalancer) {
        this.clientFactory = clientFactory;
        this.loadBalancer = loadBalancer;
    }

    @Override
    public ClientHttpRequest createRequest(URI originalUri, HttpMethod httpMethod) throws IOException {
        String serviceId = originalUri.getHost();
        IClientConfig clientConfig = clientFactory.getClientConfig(serviceId);

        RestClient client = clientFactory.getClient(serviceId, RestClient.class);
        HttpRequest.Verb verb = HttpRequest.Verb.valueOf(httpMethod.name());
        return new RibbonHttpRequest(originalUri, verb, client, clientConfig);
    }

    public class RibbonHttpRequest extends AbstractClientHttpRequest {

        private HttpRequest.Builder builder;
        private URI uri;
        private HttpRequest.Verb verb;
        private RestClient client;
        private IClientConfig config;
        private ByteArrayOutputStream outputStream = null;

        public RibbonHttpRequest(URI uri, HttpRequest.Verb verb, RestClient client, IClientConfig config) {
            this.uri = uri;
            this.verb = verb;
            this.client = client;
            this.config = config;
            this.builder = HttpRequest.newBuilder().uri(uri).verb(verb);
        }

        @Override
        public HttpMethod getMethod() {
            return HttpMethod.valueOf(verb.name());
        }

        @Override
        public URI getURI() {
            return uri;
        }

        @Override
        protected OutputStream getBodyInternal(HttpHeaders headers) throws IOException {
            if (outputStream == null) {
                outputStream = new ByteArrayOutputStream();
            }
            return outputStream;
        }

        @Override
        protected ClientHttpResponse executeInternal(HttpHeaders headers) throws IOException {
            try {
                addHeaders(headers);
                if (outputStream != null) {
                    outputStream.close();
                    builder.entity(outputStream.toByteArray());
                }
                HttpRequest request = builder.build();
                HttpResponse response = client.executeWithLoadBalancer(request, config);
                return new RibbonHttpResponse(response);
            }
            catch (Exception e) {
                throw new IOException(e);
            }

            //TODO: fix stats, now that execute is not called
            // use execute here so stats are collected
            /*
            return loadBalancer.execute(this.config.getClientName(), new LoadBalancerRequest<ClientHttpResponse>() {
                @Override
                public ClientHttpResponse apply(ServiceInstance instance) throws Exception {}
            });
            */
        }

        private void addHeaders(HttpHeaders headers) {
            for (String name : headers.keySet()) {
                // apache http RequestContent pukes if there is a body and
                // the dynamic headers are already present
                if (!isDynamic(name) || outputStream == null) {
                    List<String> values = headers.get(name);
                    for (String value : values) {
                        builder.header(name, value);
                    }
                }
            }
        }

        private boolean isDynamic(String name) {
            return name.equals("Content-Length") || name.equals("Transfer-Encoding");
        }
    }

    public class RibbonHttpResponse extends AbstractClientHttpResponse {

        private HttpResponse response;
        private HttpHeaders httpHeaders;

        public RibbonHttpResponse(HttpResponse response) {
            this.response = response;
            this.httpHeaders = new HttpHeaders();
            List<Map.Entry<String, String>> headers = response.getHttpHeaders().getAllHeaders();
            for (Map.Entry<String, String> header : headers) {
                this.httpHeaders.add(header.getKey(), header.getValue());
            }
        }

        @Override
        public InputStream getBody() throws IOException {
            return response.getInputStream();
        }

        @Override
        public HttpHeaders getHeaders() {
            return this.httpHeaders;
        }

        @Override
        public int getRawStatusCode() throws IOException {
            return response.getStatus();
        }

        @Override
        public String getStatusText() throws IOException {
            return HttpStatus.valueOf(response.getStatus()).name();
        }

        @Override
        public void close() {
            response.close();
        }
    }
}