Apache DefaultHttpClient - java.net.BindException: Address already in use: connect

Question

I am running a performance test in a Java "client" accessing a Tomcat 8.5 web server. After about 13,000 requests HTTP request fails with the error,

org.apache.http.impl.client.DefaultHttpClient tryConnect
INFO: Retrying connect
java.net.BindException: Address already in use: connect
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)

The code is,

    for (int i = 0; i < 15000; i++) {
        try {
            if (i % 1000 == 0) System.out.println("Iterations: " + Integer.toString(i));
            HttpGet request = new HttpGet("http://localhost:9080");
            DefaultHttpClient client = new DefaultHttpClient();
            HttpResponse response = client.execute(request, new BasicHttpContext());
            HttpEntity entity = response.getEntity();
            EntityUtils.consume(entity);
        } catch (Exception e) {
            e.printStackTrace();
            System.out.println("Iterations: " + Integer.toString(i));
            System.exit(1);
        }
    }

If I cache the DefaultHttpClient the error does not occur. Also tried,

        request.releaseConnection();
        client.getConnectionManager().shutdown();

But does not change the error. The error does not seem to be cause by the client. If I access another website in the URL it is okay. Seems to be caused by Tomcat in Windows running out of file handles or socket resources or something. If I run it again as another process right after it crashes, it will fail in 1 run not 13,000, so the issue is Tomcat running out of resources. Seems like DefaultHttpClient is not closing its connection, or Tomcat is not freeing its connections until a gc occurs.

Using HTTPClient 4.2.5

Any ideas why it occurs, or how to fix?

Answer 1

I couldn’t reproduce the same error you have. Anyways when I ran your example in single thread I receive NoRouteToHostException.

13:37:57.917 [main] DEBUG org.apache.http.impl.conn.BasicClientConnectionManager - Releasing connection org.apache.http.impl.conn.ManagedClientConnectionImpl@2c7ceffa
Iterations: 16329
java.net.NoRouteToHostException: Can't assign requested address (Address not available)
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
        at java.base/java.net.Socket.connect(Socket.java:591)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:605)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:440)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at com.demo.DemoApplication.main(DemoApplication.java:25)

You are opening connections at the rate faster than it takes to close the connection - Closing a socket is held in TIME_WAIT status before they are released to be available for new connections.

For testing purposes only you can set the tcp_tw_reuse to allow TIME_WAIT sockets to be reused.

On my OSX I could change the max segment lifetime for testing and the error disappeared.

sudo sysctl -w net.inet.tcp.msl=1000 net.inet.tcp.msl: 15000 -> 1000

DefaultHttpClient is backed by BasicClientConnectionManager which create and manage single connection and keeps only one active connection at a time for any route.

When the connection is released back to connection manager it is kept alive for reuse and is marked as reusable.

If I cache the DefaultHttpClient the error does not occur.

That is exactly the solution. I believe correct way to be use single http client and allow the connection managers to do its job.

All the connection management is explained in apache documentation

4.2.5 is pretty old (April 2013). If your starting on a new project it would make sense to update to latest(4.5.12) at the time of writing.

References:

https://cwiki.apache.org/confluence/display/HTTPCOMPONENTS/FrequentlyAskedConnectionManagementQuestions

https://raby.sh/osx-where-are-my-time_wait.html

Answer 2

Best Practice is to use MultiThread HttpClient. Below should work :

package com.demo.httpclient;

import java.net.URI;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.impl.conn.DefaultClientConnection;
import org.apache.http.impl.conn.PoolingClientConnectionManager;
import org.apache.http.util.EntityUtils;

public class ThreadedHttpClientTest {

    private static URI rootUri = URI.create("http://localhost:8080/");
    private static int worker = 100;
    private static int count = 15000;

    private static HttpClient httpClient;

    public static void main(String[] args) throws Exception {

        long startTime = System.currentTimeMillis();
        PoolingClientConnectionManager poolingClientConnectionManager = new PoolingClientConnectionManager();
        poolingClientConnectionManager.setMaxTotal(worker);
        poolingClientConnectionManager.setDefaultMaxPerRoute(worker);

        httpClient = new DefaultHttpClient(poolingClientConnectionManager);

        List<Callable<Void>> workers = new ArrayList<Callable<Void>>();

        for (int i = 0; i < count; i++) {
            workers.add(new WorkerThread(httpClient, rootUri.toString()));
        }

        ExecutorService pool = Executors.newFixedThreadPool(worker);

        int i=0;
        for (Future<Void> future : pool.invokeAll(workers)) {
            future.get();
            System.out.println("Response " + i++);
        }

        System.out.println("Time Taken :: " + (System.currentTimeMillis() - startTime) + "ms");
        pool.shutdown();
    }

    static class WorkerThread implements Callable<Void> {

        HttpClient client;
        String url;

        public WorkerThread(HttpClient httpClient, String url) {
            this.client = httpClient;
            this.url = url;
        }

        @Override
        public Void call() throws Exception {
            HttpGet get = new HttpGet(url);
            HttpResponse response = client.execute(get, new DefaultClientConnection());
            HttpEntity entity = response.getEntity();
            EntityUtils.consume(entity);
            return null;
        }
    }
}

Answer 3

It seems like you are using HttpClient with a wrong way (i.e probably copy-paste old legacy snippet)

• First of all one client is enough (for one thread).
• According to the httpcomponents-client documentation it's better to use pooled connection management
• Your response should be closed

I.e.

  PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
  CloseableHttpClient client = HttpClients.custom()
          .setConnectionManager(cm)
          .build();
  HttpContext context = HttpClientContext.create();
  try {
    final HttpGet request = new HttpGet("http://localhost:9080");
    for(int i = 0; i < 15000; i++) {
      CloseableHttpResponse response = httpClient.execute(request, context);
      try {
        HttpEntity entity = response.getEntity();
        EntityUtils.consume(entity);
      } finally {
        response.close();
      }
    }
  } finally {
    client.close();
  }
}