Apache HTTPClient 4.x vs multiple JDK URL connections

Question

I am trying out Apache HTTPClient 4.3.6 connection pool manager to increase the throughput of my HTTP calls. My assumption is, HTTPClient implementation in general is using persistence connection. The result of my test code (included at the end), however, showed that multiple concurrent HTTP connections using JDK URLConnection perform better.

How do I make HTTPClient fast?
Does HTTPClient uses the same HTTP connection for http://localhost:9000/user/123 and http://localhost:9000/user/456?

Thanks

import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;

public class FooTest {

public static void main(String[] args) throws Exception {
    runWithConnectionPool();
}

 private static String extract(BufferedReader reader) throws Exception {
     StringBuilder buffer = new StringBuilder();
     String line = null;
     while ((line = reader.readLine()) != null) {
         buffer.append(line);
     }
     return buffer.toString();
 }

 private static void runWithConnectionPool() throws Exception {
     PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
     cm.setMaxTotal(1);

     CloseableHttpClient httpClient = HttpClients.custom()
         .setConnectionManager(cm)
         .setMaxConnTotal(100)
         .setMaxConnPerRoute(100)
         .build();

     long start = System.currentTimeMillis();

     HttpGet getReq = new HttpGet("http://www.google.com");

     /*
       Option A: Using HTTP connection pool
       Option B: Individual JDK 8 URL connection
     */
 //    Thread[] workers = generateAndStart(10, httpClient, getReq, 0);                 // (A)
     Thread[] workers = generateAndStart(10, getReq.getURI().toURL(), 0);  // (B)

     for (int i = 0; i < workers.length; i++) {
         workers[i].join();
     }

     System.out.println("Elasped: " + (System.currentTimeMillis() - start));
 }

 private static Thread[] generateAndStart(int num, URL url, long delay) {
     Thread[] workers = new Thread[num];
     for (int i = 0; i < num; i++) {
         System.out.println("Starting worker: " + i);
         int j = i;
         workers[i] = new Thread(() -> connect(url, delay, j));
         workers[i].start();
     }
     return workers;
 }

 private static void connect(URL url, long delay, int ndx)  {
     try {
         System.out.println(url.toURI().toString() + " started.");
     } catch (Exception e) {
         e.printStackTrace();
     }

     try {

         URLConnection connection = url.openConnection();
         connection.addRequestProperty("Accept", "application/json");
         BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));

         String line;
         while ((line = reader.readLine()) != null) {
             ObjectMapper mapper = new ObjectMapper();
             System.out.println(line);
         }
         if (delay > 0) {
             System.out.println("Delayed.");
             sleep(delay);
         }
         reader.close();
     } catch (Exception e) {
         e.printStackTrace();
     }
 }

 private static Thread[] generateAndStart(int num, CloseableHttpClient httpClient, HttpGet getReq, long delay) {
     Thread[] workers = new Thread[num];
     for (int i = 0; i < num; i++) {
         System.out.println("Starting worker: " + i);
         final int j = i;
         workers[i] = new Thread(() -> connect(httpClient, getReq, delay, j));
         workers[i].start();
     }
     return workers;
 }

 private static void connect(CloseableHttpClient httpClient, HttpGet request, long delay, int ndx) {
     System.out.println(request.getURI().toString() + " started.");

     try(
         CloseableHttpResponse response = httpClient.execute(request, HttpClientContext.create());
         BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent()))) {

         String line;
         while ((line = reader.readLine()) != null) {
             ObjectMapper mapper = new ObjectMapper();
             System.out.println(line);
         }
         if (delay > 0) {
             System.out.println("Delayed.");
             sleep(delay);
         }
     } catch (Exception e) {
         e.printStackTrace();
     }
 }

 private static void sleep(long delay) {
     try {
         Thread.sleep(delay);
     } catch (Exception e) {
         e.printStackTrace();
     }
 }
 }

Update 1 (28 March 2017)

I have made several observations and conclusions,

JDK java.net.URLConnection does not make a connection until URLConnection.getInputStream() is called.
java.net.URLConnection closes the current socket if bad connection happens e.g. HTTP error, and creates a new socket.
Using the java.net.URLConnection instances created from the same java.net.URL instance in a multi threaded environment will create multiple sockets to the server. Instead, for simplicity, invoke URL.openConnection() in a synchronized block. ~~On the same note, it does not mean every call to the URL.openConnection() will create a new socket. I believe URL regulates this.~~ I was wrong. The number of sockets created is according to the number of threads that invoke URL.openConnection().
It is mentioned in many places and I mention again, closing / disconnecting the URLConnection does not close the socket.
Connecting to a different path of the same server does not create another connection socket. In other words, persistent connection is usable with different paths.
Apache HTTPClient in general is easier to use and more intuitive. It supports persistent connection (uses the same socket for connection) in a multi threaded environment without user's intervention.
I could not get URL to comply to http.maxConnections and http.keepAlive. For example, I include -Dhttp.keepAlive=false during runtime does not prevent Connection: keep-alive included in the HTTP headers.

My observations come from the examples I pasted here. They are better examples than the code pasted above.

Apache HTTPClient 4.x vs multiple JDK URL connections

Update 1 (28 March 2017)

Answers (1)

Related Questions