How to set TCP keep Alive from HttpClient?

Question

My Java application which resides in AWS private subnet connects to an http server via AWS Nat gateway. I am calling a POST request via HttpClient to the HTTP server. That request will take more than 10 minutes to complete. I have configured a socket time out and connection timeout of 1 hour as this this a background task . But the intermediate AWS NAT gateway will send back a RST packet after 300 secs [5 mins] and cause the connection to get resetted , there is no way i can increase the NAT gateway timeout. So i need to handle the problem from my application side.

My strategy is to use a TCP keep alive time which will send a packet say every 240 secs to keep the connection active. I have configured this as below

CloseableHttpClient httpClient = HttpClients.createDefault()
HttpParams params = httpClient.getParams();
HttpConnectionParams.setConnectionTimeout(params, 3600000); //connection Timeout
HttpConnectionParams.setSoTimeout(params, 3600000); // Socket Time out
HttpConnectionParams.setSoKeepalive(params, true); //Enable Socket level keep alive time

and then call the post request via execute method

HttpPost post = new HttpPost("http://url");
HttpResponse response = httpClient.execute(post);

Since I am using a Linux system I have configured the server with following sysctl values:

sysctl -w net.ipv4.tcp_keepalive_time=240 
sysctl -w net.ipv4.tcp_keepalive_intvl=240
sysctl -w net.ipv4.tcp_keepalive_probes=10

But while executing the program the keep alive is not enabled and connection fails as previous.

I have checked this with netstat -o option and as shown below keep alive is off

tcp        0      0 192.168.1.141:43770     public_ip:80          ESTABLISHED 18134/java           off (0.00/0/0)

Is there any way i can set TCP keep alive from java code using httpclient . Also I can see HttpConnectionParams are deprecated. But I couldn't find any new class which can set keep alive

Answer 1

Sometimes, if the configuration is overwritten, the configuration does not take effect.My initial modification of setDefaultSocketConfig in buildClient didn't take effect.Because it is overwritten by getConnectionManager()

    public CloseableHttpClient buildClient() throws Exception {
    HttpClientBuilder builder = HttpClientBuilder.create()
            .setDefaultSocketConfig(SocketConfig.custom().setSoKeepAlive(true).build())  // did not work
            .setConnectionManager(getConnectionManager())
            .setRetryHandler(getRequestRetryHandler())
            .setConnectionReuseStrategy(getConnectionReuseStrategy())
            .setDefaultConnectionConfig(getConnectionConfig())
            .setDefaultRequestConfig(getRequestConfig())
            .setDefaultHeaders(getDefaultHeaders())
            .setDefaultCredentialsProvider(getDefaultCredentialsProvider())
            .disableContentCompression() // gzip is not needed. Use lz4 when compress=1
            .setDefaultCookieStore(cookieStoreProvider.getCookieStore(properties))
            .disableRedirectHandling();

    String clientName = properties != null ? properties.getClientName() : null;
    if (!Utils.isNullOrEmptyString(clientName)) {
        builder.setUserAgent(clientName);
    }
    
    return builder.build();

And then I move the config to getConnectionManager()，and it work.

    private PoolingHttpClientConnectionManager getConnectionManager()
    throws CertificateException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException, IOException {
    RegistryBuilder<ConnectionSocketFactory> registry = RegistryBuilder.<ConnectionSocketFactory>create()
      .register("http", PlainConnectionSocketFactory.getSocketFactory());

    if (properties.getSsl()) {
        HostnameVerifier verifier = "strict".equals(properties.getSslMode()) ? SSLConnectionSocketFactory.getDefaultHostnameVerifier() : NoopHostnameVerifier.INSTANCE;
        registry.register("https", new SSLConnectionSocketFactory(getSSLContext(), verifier));
    }

    //noinspection resource
    PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager(
        registry.build(),
        null,
        null,
        new IpVersionPriorityResolver(),
        properties.getTimeToLiveMillis(),
        TimeUnit.MILLISECONDS
    );

    connectionManager.setDefaultMaxPerRoute(properties.getDefaultMaxPerRoute());
    connectionManager.setMaxTotal(properties.getMaxTotal());
    connectionManager.setDefaultConnectionConfig(getConnectionConfig());
    connectionManager.setDefaultSocketConfig(SocketConfig.custom().setSoKeepAlive(true).build());
    return connectionManager;
}

Answer 2

The approach above with Socket worked beautifully with a reset of tcp_keepalive_intvl value below the AWS Network Load Balancer timeout. Using both, reset the NLB tcp idle timeout that allowed java hour+ connections.

Answer 3

Keeping an HTTP connection open but inactive for a long period is a bad design choice. HTTP is a request-response protocol, with the implication that requests and responses are quick.

Holding a connection open holds resources. From the perspective of the server (and network firewalls and routers) a client that opens a connection and begins a request (A POST in your case) but does not send any bytes for a long period is indistinguishable from a client that will never send any more data, because it is faulty or malicious (conducting a DOS attack). The server (and network hardware) is right to conclude that the right thing to do is to shutdown the connection and reclaim the resources used for it. You are trying to fight against correct behaviour that occurs for good reasons. Even if you manage to workaround the TCP shutdown you will find other problems, such as HTTP server timeouts and database timeouts.

You should instead be reconsidered the design of communication between the two components. That is, this looks like an XY Problem. You might consider

• Having the client wait until it has a complete upload to perform before starting the POST.
• Splitting the uploads into smaller, more frequent uploads.
• Use a protocol other than HTTP.

Answer 4

I have found a solution to the problem . Curious case is there is no way i can use some builder class in httpclient to pass socket keep alive . One method as i specified in the question is using HttpConnectionParams as below but this is not working and this class is now deprecated.

HttpParams params = httpClient.getParams();
HttpConnectionParams.setSoKeepalive(params, true);

So while checking apache http docs I can see that now connection parameters are passed to httpclient via RequestConfig class . Builders of this class provide solution to set connection_time_out and socket_time_out. But checking the socurce code of this I couldnt see an option to enable SocketKeepAlive which is what we want. So the only solution is directly creating a Socket using SocketBuilder class and pass that to the HttpClientBuilder.

Following is the working code

SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true).setSoTimeout(3600000).build(); //We need to set socket keep alive
        RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(3600000).build();
        CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).
                                           setDefaultSocketConfig(socketConfig).build();
HttpPost post = new HttpPost(url.toString());
HttpResponse response = httpClient.execute(post);

While executing above i can see that keep alive is properly set in the socket based on the sysctl values i set in linux kernel

tcp        0      0 localip:48314     public_ip:443     ESTABLISHED 14863/java          keepalive (234.11/0/0)

If some one has a better solution to enable Socket Keep alive from Requestconfig class or any other high level builder class i am open to suggestions.