CODEWITHSUNDEEP
javaapacheapache-httpclient-4.x
shagun
shagun

Reputation: 249

How to get httpclient response as stream

I am using httpclient 4.5.5 i want to get large files upto 1 gb in response. But it seems like

CloseableHttpResponse response = httpClient.execute(httpGet); HttpEntity entity = response.getEntity();

This returns whole response so its not good to have whole response in memory. Is there any way to get response as stream?

Upvotes: 7

Views: 20703

Answers (3)

ok2c
ok2c

Reputation: 27613

Apache HttpClient as of version 4.0 (as well as Apache HttpAsyncClient) supports full content streaming for both incoming and outgoing HTTP messages. Use HttpEntity to get access to the underling content input stream 

CloseableHttpClient client = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://myhost/tons-of-stuff");
try (CloseableHttpResponse response1 = client.execute(httpGet)) {
    final HttpEntity entity = response1.getEntity();
    if (entity != null) {
        try (InputStream inputStream = entity.getContent()) {
            // do something useful with the stream
        }
    }
}

Upvotes: 17

Felix Ng
Felix Ng

Reputation: 612

You need Apache Async Client.

HttpAsyncClient is the ASYNC version of Apache HttpClient. Apache HttpClient construct the whole response in memory, while with HttpAsyncClient, you can define a Handler (Consumer) to process the response while receiving data.

https://hc.apache.org/httpcomponents-asyncclient-4.1.x/index.html

Here is an example from their official example code

package org.apache.http.examples.nio.client;

import java.io.IOException;
import java.nio.CharBuffer;
import java.util.concurrent.Future;

import org.apache.http.HttpResponse;
import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
import org.apache.http.impl.nio.client.HttpAsyncClients;
import org.apache.http.nio.IOControl;
import org.apache.http.nio.client.methods.AsyncCharConsumer;
import org.apache.http.nio.client.methods.HttpAsyncMethods;
import org.apache.http.protocol.HttpContext;

/**
 * This example demonstrates an asynchronous HTTP request / response exchange with
 * a full content streaming.
 */
public class AsyncClientHttpExchangeStreaming {

    public static void main(final String[] args) throws Exception {
        CloseableHttpAsyncClient httpclient = HttpAsyncClients.createDefault();
        try {
            httpclient.start();
            Future<Boolean> future = httpclient.execute(
                    HttpAsyncMethods.createGet("http://httpbin.org/"),
                    new MyResponseConsumer(), null);
            Boolean result = future.get();
            if (result != null && result.booleanValue()) {
                System.out.println("Request successfully executed");
            } else {
                System.out.println("Request failed");
            }
            System.out.println("Shutting down");
        } finally {
            httpclient.close();
        }
        System.out.println("Done");
    }

    static class MyResponseConsumer extends AsyncCharConsumer<Boolean> {

        @Override
        protected void onResponseReceived(final HttpResponse response) {
        }

        @Override
        protected void onCharReceived(final CharBuffer buf, final IOControl ioctrl) throws IOException {
            while (buf.hasRemaining()) {
                System.out.print(buf.get());
            }
        }

        @Override
        protected void releaseResources() {
        }

        @Override
        protected Boolean buildResult(final HttpContext context) {
            return Boolean.TRUE;
        }

    }

}

Upvotes: 5

Kosala Samarasinghe
Kosala Samarasinghe

Reputation: 39

Use HttpURLConnection instead of httpClient.

final HttpURLConnection conn = (HttpURLConnection)url.openConnection();
final int bufferSize = 1024 * 1024;
conn.setChunkedStreamingMode(bufferSize);
final OutputStream out = conn.getOutputStream();

Upvotes: 1

Related Questions