CODEWITHSUNDEEP
spring-integration
Sujit
Sujit

Reputation: 476

Timed out waiting for response exception at TcpOutboundGateway at Spring TCP Integration

Currently I am working on an external vendor integration via Spring Integration TCP where I am sending to different ports of 2 IPs (IP 1, port 1, 2 ,3 and IP 2 port 1 , 2 & 3) by creating different @ServiceActivator for unique IP/Port combination.

All is working fine in a normal messages load, however, when I increase the load to 40-50 transactions/sec, then I am seeing the messages are getting timeout by waiting for the response.

Below is one of the Service Activator and others are setup in the similar way as well. I am using FailoverClientConnectionFactory for failover purpose if my primary ip is not available. I am also using CachingClientConnectionFactory so, that every request I don't have to create a connection which is expensive.

@Bean
@ServiceActivator(inputChannel = "toRequest")
public MessageHandler serviceActivatorOne() {
    TcpOutboundGateway tcpOutputGateway = new TcpOutboundGateway();
    List<ServerNode> nodes = properties.getFailOver().getServers().subList(0,2);
    tcpOutputGateway.setConnectionFactory(createFailOverConnectionFactory(nodes));
    tcpOutputGateway.setReplyChannelName("bytesToObjectChannel");
    tcpOutputGateway.setRemoteTimeout(5000);
    return tcpOutputGateway;
}

...
...

private AbstractClientConnectionFactory createFailOverConnectionFactory(List<ServerNode> serverNodeList) {
        FailoverClientConnectionFactory connectionFactory =
                new FailoverClientConnectionFactory(createConnectionFactories(serverNodeList));
        connectionFactory.setSingleUse(true);
        connectionFactory.afterPropertiesSet();
        return connectionFactory;
    }

private AbstractClientConnectionFactory createDefaultConnectionFactory(String url, int port, String name) {
        TcpNetClientConnectionFactory connFactory = new TcpNetClientConnectionFactory(url, port);
        connFactory.setSerializer(byteArrayLengthHeaderSerializer());
        connFactory.setDeserializer(byteArrayLengthHeaderSerializer());
        connFactory.setSoTimeout(properties.getSocketTimeOut());
        connFactory.setSoKeepAlive(true);
        connFactory.setLookupHost(false);
        connFactory.setComponentName(name);
        connFactory.setSingleUse(true);

        if(properties.getPoolMaxSize() <= 0) {
            return connFactory;
        } else {
            CachingClientConnectionFactory cachingConnFactory =
                    new CachingClientConnectionFactory(connFactory, properties.getPoolMaxSize());
            cachingConnFactory.setConnectionWaitTimeout(properties.getPoolMaxWait());
            return cachingConnFactory;
        }
    }

Exception details:

[DEBUG]-[http-nio-9090-exec-103]-[11-Nov-2019;22:01:14.403]-[TcpOutboundGateway:handleRequestMessage:162]-Remote Timeout on 1980e8d4-3167-4610-b8ac-8fb0d20eb92a:1
[DEBUG]-[http-nio-9090-exec-103]-[11-Nov-2019;22:01:14.403]-[SimplePool:releaseItem:226]-Releasing TcpNetConnection:216.76.27.251:20303:54340:d9c78e03-ecbb-45af-b502-6aff9a9b0036 back to the pool
[ERROR]-[http-nio-9090-exec-103]-[11-Nov-2019;22:01:14.404]-[TcpOutboundGateway:handleRequestMessage:174]-Tcp Gateway exception
org.springframework.integration.MessageTimeoutException: Timed out waiting for response
    at org.springframework.integration.ip.tcp.TcpOutboundGateway.handleRequestMessage(TcpOutboundGateway.java:166)
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:123)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:162)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:144)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:105)
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:73)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:453)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:401)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
    at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:4

I tried to find out some details from Spring reference on TCP but didn't succeed so far. I am not using shared socket as setSingleUse is set as false. Any help/clue is appreciated.

Upvotes: 1

Views: 2577

Answers (1)

Artem Bilan
Artem Bilan

Reputation: 121272

It looks like the server you connect cannot answer in time for that 5 seconds timeout under heavy load.

It seems the default remoteTimeout is 10s and I am setting to 5s which seems the server can't response with in 5s. I can tune the remoteTimeout and check. Do you have any suggestions where I can change any of the spring-tcp integration settings or is it solely depends on slow responsive nature from the server?

Right, it does depend on server. I imagine a situation when with some huge load even those 10 seconds won’t be enough. You should consult with your server vendor to determine their throughput.

Upvotes: 1

Related Questions