CODEWITHSUNDEEP
javaspringresttemplate
petronius
petronius

Reputation: 489

RestTemplate: Is there a way to protect jvm agains huge response size?

When using a RestTemplate to talk to an external service, I've seen more than once OutOfMemory errors on our application because the service streams gigs of data (due to a bad implementation on their side, in case of errors they they were sending back big stacktraces in each element of the array, which usually contains few thousand). It was ending in about 6gb of data, serialized by jackson in our app and totally exploding the Xmx of the jvm.

I've looked around but there don't seem to be any way to protect against this kind of even, i.e. aborting the request when the streamed response exceed a given size.

Is there a solution to this? We are using apache's httpcomponents httpclient 4.5.5, but any other underlying implementation would be acceptable.

Besides RestTemplates, a solution for Spring's reactive WebClient would also be welcome.

Upvotes: 2

Views: 872

Answers (2)

petronius
petronius

Reputation: 489

For the records, here is the final solution. The problematic was to load a list of objects that can be very big, using pagination (through elastic search scroll api).

ResponseExtractor<Car[]> responseExtractor = responseEntity -> {
    long pageContentLengthInBytes = responseEntity.getHeaders().getContentLength();
    long presumableFreeMemoryInBytes = this.getAvailableFreeMemoryAmount();

    if (presumableFreeMemoryInBytes - TWENTY_MEGABYTES < pageContentLengthInBytes) {
        log.error("Not enough memory to store the page ({} avaiable, content-length={}, trashing it", presumableFreeMemoryInBytes, pageContentLengthInBytes);
        responseEntity.close();
        return null;
    }
    return objectMapper.readValue(responseEntity.getBody(), Car[].class);
};

Car[] responseEntities = this.restTemplate.execute(uri, HttpMethod.GET, null, responseExtractor);

   /**
     * Returns the current amount of memory which may be allocated until an out-of-memory error occurs.
     * see https://stackoverflow.com/a/12807848/8836232
     */
    private long getAvailableFreeMemoryAmount() {
        long allocatedMemory = (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory());
        return Runtime.getRuntime().maxMemory() - allocatedMemory;
    }

Upvotes: 0

CodeScale
CodeScale

Reputation: 3304

This has to be enforced at the underlying HTTP client library (spring supports different ones like JDK client, apache client, okHTTP..)

Here you talking about apache-httpcomponent , did you check this HttpEntity.getContent() ? It actually returns an InputStream that you can read by yourself and determine when the size has been exceeded..

https://hc.apache.org/httpcomponents-core-4.4.x/httpcore/apidocs/org/apache/http/HttpEntity.html

Upvotes: 1

Related Questions