CODEWITHSUNDEEP
javahttpurlconnection
parsecer
parsecer

Reputation: 5106

HttpURLConnection getInputStream is too slow

I have a site with an API which I use to get JSON data. I have a ConnectClass class, an instance of which is created each time a request is sent. When an instance of ConnectClass is created, a new HttpURLConnection object is created, .setup()ed and .connect()ed:

class ConnectClass  {
        private HttpURLConnection connection;

        private String link;
        private REQUEST_TYPE requestType;
        private URL url;
        private AccessToken accessToken;  

        public String send() throws Exception  {
                connection.connect();
                System.out.println("start get input stream");  //from this
                InputStream input = connection.getInputStream();

                System.out.println("end get input stream");  //to this takes too long

                System.out.println("Start scanner");
                String inputString = new Scanner(input, "UTF-8").useDelimiter("\\Z").next();
                System.out.println("End scanner");
                input.close();

                return inputString;  //returns response JSON string
        }

        public ConnectClass(String link, AccessToken accessToken, REQUEST_TYPE requestType)  throws Exception {
            this.link = link;
            this.accessToken = accessToken;
            this.requestType = requestType;

            this.url = new URL(link);
            connection = (HttpURLConnection)  url.openConnection();
            setup();
        }

        private void setup() throws Exception {
            //connection.setDoOutput(true);
            connection.setConnectTimeout(100);  //doesn't really change things 
            //connection.setChunkedStreamingMode(1024);
            connection.setRequestProperty("charset", "utf-8");
            connection.setRequestProperty("User-Agent", "some description v2.0");

            connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");

            connection.setRequestProperty("Authorization", "Bearer " + accessToken.ACCESS_TOKEN_LONG);
            connection.setRequestMethod("GET");

            connection.setInstanceFollowRedirects(false);
        }
    }

However, I need to send 10 requests like this in a loop. And each request takes about 1.3 seconds, when all 10 requests should take no more than 1-2 seconds combined. I've found most time is spend on getting input stream and processing it: InputStream input = connection.getInputStream(); - it takes about 0.6-1 second and String inputString = new Scanner(input, "UTF-8").useDelimiter("\\Z").next(); takes about 0.1-0.2 seconds.

Is there anything I can do to lower the time per request?

I tried setting connect timeout to as low as 100, but it didn't have a visible effect.

EDIT: Response JSON is rather big. connection.setRequestProperty("Accept-Encoding", "gzip"); and then using InputStream input = new GZIPInputStream(connection.getInputStream()); helps, but it only saves about 4-5 seconds in total.

I can't use concurrent requests - every new request depends on the previous one (takes a parameter from previous input JSON and passes it in a new request, in a link).

Upvotes: 1

Views: 3909

Answers (1)

David Soroko
David Soroko

Reputation: 9086

Bellow is your code with bits removed to allow compilation. Main doing a GET on Google. On my machine send takes ~ 300 millis. to complete. If you are getting a similar timing, I would suggest that you check that your account has been set up properly and is not throttled for example

import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Scanner;

class ConnectClass {
    private HttpURLConnection connection;

    private String link;
    private URL url;

    public String send() throws Exception {
        connection.connect();
        System.out.println("start get input stream");  //from this
        InputStream input = connection.getInputStream();

        System.out.println("end get input stream");  //to this takes too long

        System.out.println("Start scanner");
        String inputString = new Scanner(input, "UTF-8").useDelimiter("\\Z").next();
        System.out.println("End scanner");
        input.close();


//        System.out.println(inputString);
        return inputString;  //returns response JSON string
    }

    public ConnectClass(String link) throws Exception {
        this.link = link;

        this.url = new URL(link);
        connection = (HttpURLConnection) url.openConnection();
        setup();
    }

    private void setup() throws Exception {
        connection.setDoOutput(true);
        connection.setConnectTimeout(100);  //doesn't really change things
        connection.setRequestProperty("charset", "utf-8");
        connection.setRequestProperty("User-Agent", "some description v2.0");
        connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");

        connection.setRequestMethod("GET");
        connection.setInstanceFollowRedirects(false);
    }


    public static void main(String[] args) throws Exception {

        ConnectClass cc = new  ConnectClass("https://www.google.com");

        long start = System.currentTimeMillis();
        cc.send();
        System.out.println("Done in " + (System.currentTimeMillis() - start));
    }
}

Another possibility is that you are receiving a large response. If so you may want to use an HTTP client that supports compression.

Upvotes: 1

Related Questions