Download large file through Spring rest template

Question

Server Code :

@POST
@Path("reportDownload")
@Consumes(MediaType.APPLICATION_JSON)
public Response generateReport(QueryData queryData) {
     File file = new File("report.xlsx") // large file
     StreamingOutput stream = new FileStreamingOutput(file) ; 
        return Response.ok(stream, MediaType.APPLICATION_OCTET_STREAM)
            .header("filename" , file.getName())
            .build();
}

Client Code :

Using the following code I'm able to download files upto some limit. Getting out of memory heap error for large files.

final String uri = buildUri("/reportGenerate/reportDownload");

    HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
    factory.setReadTimeout(read_timeout);
    factory.setConnectTimeout(connection_timeout);

    RestTemplate restTemplate = new RestTemplate(factory);

    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.APPLICATION_JSON);
    List<MediaType> mediaTypeList = new ArrayList<>();
    mediaTypeList.add(MediaType.APPLICATION_OCTET_STREAM);
    headers.setAccept(mediaTypeList);
    HttpEntity entity = new HttpEntity(queryData, headers);
    ResponseEntity<byte[]> data = restTemplate.exchange(uri, HttpMethod.POST, entity, byte[].class);
    HttpHeaders responseHeader = data.getHeaders();
    String fileName = (String) responseHeader.get("filename").get(0);
    String downloadFolder = ApplicationConfig.REPORT_DOWNLOAD_FOLDER.getValue();
    if (data.getStatusCode() == HttpStatus.OK) {
        FileOutputStream fos = null;
        File toFile = null;
        try {
            toFile = new File(downloadFolder + File.separator + fileName);
            fos = new FileOutputStream(toFile);
            ByteArrayOutputStream bos = new ByteArrayOutputStream();
            IOUtils.write(data.getBody(), bos);
            bos.writeTo(fos);

        } catch (Exception e) {
            convertReportException(e);
        } finally {
            if (fos != null) {
                try {
                    fos.close();
                } catch (IOException ex) {
                    convertReportException(ex);
                }
            }
            return toFile;
        }
    }

How to use stream for download larger files.

Answer 1

At the client that you mentioned
don't store the file in memory to download a large via the RestTemplate, it can cause the Java heap exception.
it should be stored on disk.

Here is some code sample to download a large file via the RestTemplate

@GetMapping("largeFile")
    public ResponseEntity<InputStreamResource> downloadLargeFile(
            @RequestParam("fileName") String fileName
    ) throws IOException {

        RestTemplate restTemplate = new RestTemplate();

        // Optional Accept header
        RequestCallback requestCallback = request -> request.getHeaders()
                .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

        // Streams the response instead of loading it all in memory
        ResponseExtractor<InputStreamResource> responseExtractor = response -> {
            // Here I write the response to a file but do what you like
            Path path = Paths.get("tmp/" + fileName);
            Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING);
            return new InputStreamResource(new FileInputStream(String.format("tmp/%s", fileName)));
        };

        InputStreamResource response = restTemplate.execute(
            String.format("http://%s:%s/file/largeFileRestTemplate?fileName=%s", host, "9091", fileName),
            HttpMethod.GET,
            requestCallback,
            responseExtractor
        );

        return ResponseEntity
            .ok()
            .header(HttpHeaders.CONTENT_DISPOSITION, String.format("attachment; filename=%s", fileName))
            .body(response);
    }

Answer 2

Here is how I do it with a ResponseExtractor. Based on hints from this Spring Jira issue.

RestTemplate restTemplate // = ...;

// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
        .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
    // Here I write the response to a file but do what you like
    Path path = Paths.get("some/path");
    Files.copy(response.getBody(), path);
    return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);

update

Here is what RestTemplate does behind the scenes on postForObject and friends (inline comments from me):

@Override
public <T> T postForObject(String url, Object request, Class<T> responseType, Map<String, ?> uriVariables)
        throws RestClientException {

    // From RequestCallback's javadoc:
    // Callback interface for code that operates on a ClientHttpRequest.
    // Allows to manipulate the request headers, and write to the request body. 
    //
    // Used internally by the RestTemplate, but also useful for application code.
    RequestCallback requestCallback = httpEntityCallback(request, responseType);

    // HttpMessageConverterExtractor checks the response type header and requested
    // responseType class to select the proper message converter to handle the response.
    // It also implements ResponseExtractor.
    HttpMessageConverterExtractor<T> responseExtractor =
            new HttpMessageConverterExtractor<T>(responseType, getMessageConverters(), logger);
    return execute(url, HttpMethod.POST, requestCallback, responseExtractor, uriVariables);
}

/**
 * Returns a request callback implementation that writes the given object to the
 * request stream.
 */
protected <T> RequestCallback httpEntityCallback(Object requestBody, Type responseType) {
    return new HttpEntityRequestCallback(requestBody, responseType);
}

Note: This is essentially a duplicate of my answer at https://stackoverflow.com/a/38664475/1030527 but I can't mark the questions as duplicate since neither this one or that one have upvoted answers.