Large outbound payload getting stuck in Spring integration Gateway

Question

We have a couple of Spring outbound SOAP WebServices clients (Talking to external WS). We talk to them using Spring Integration Gateway.

Versions using:

Spring Integration version - 4.0.4.RELEASE
Spring version - 4.1.6.RELEASE
Spring WS - 2.2.4
Apache http client - 4.5.1

All works fine when the payload is a few kbs. But when we have a slightly bigger payload (Say 700KB, not huge I know), the processing gets stuck. The thread dump gives the following StackTrace. This happens in both Weblogic(12C) and Apache Tomcat, but only on Linux.

Has somebody encountered this issue before? Is there some configuration inside Spring Integration/Gateway where we can set the writebuffer size?

 java.lang.Thread.State: RUNNABLE
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
        at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:864)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:835)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
        - locked <0x00000000c31b1d00> (a sun.security.ssl.AppOutputStream)
        at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:126)
        at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:162)
        at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:115)
        at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:116)
        at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:158)
        at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:162)
        at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:237)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.springframework.ws.transport.http.HttpComponentsConnection.onSendAfterWrite(HttpComponentsConnection.java:121)
        at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:48)
        at org.springframework.ws.client.core.WebServiceTemplate.sendRequest(WebServiceTemplate.java:658)
        at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:606)
        at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:555)
        at org.springframework.integration.ws.MarshallingWebServiceOutboundGateway.doHandle(MarshallingWebServiceOutboundGateway.java:81)
        at org.springframework.integration.ws.AbstractWebServiceOutboundGateway.handleRequestMessage(AbstractWebServiceOutboundGateway.java:167)
        at org.springframework.integration.handler.AbstractReplyProducingMessageHandler$AdvisedRequestHandler.handleRequestMessage(AbstractReplyProducingMessageHandler.java:313)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.integration.handler.advice.AbstractRequestHandlerAdvice$1.cloneAndExecute(AbstractRequestHandlerAdvice.java:92)
        at org.springframework.integration.handler.advice.RequestHandlerRetryAdvice$2.doWithRetry(RequestHandlerRetryAdvice.java:88)
        at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:263)
        at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:193)
        at org.springframework.integration.handler.advice.RequestHandlerRetryAdvice.doInvoke(RequestHandlerRetryAdvice.java:85)
        at org.springframework.integration.handler.advice.AbstractRequestHandlerAdvice.invoke(AbstractRequestHandlerAdvice.java:69)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
        at com.sun.proxy.$Proxy92.handleRequestMessage(Unknown Source)
        at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.doInvokeAdvisedRequestHandler(AbstractReplyProducingMessageHandler.java:189)
        at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:173)
        at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
        at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
        at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
        at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
        at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
        at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:255)
        at org.springframework.integration.handler.MessageHandlerChain$ReplyForwardingMessageChannel.send(MessageHandlerChain.java:263)

UPDATE I modified the HttpClient's DefaultSocketConfig (To a custom one) to include setSndBufSize to a large on. Still no effect!

On running netstat -ant | grep 443 the tcp status is CLOSE_WAIT (And sometimes LAST_ACK)and stays there hanging forever.

Answer 1

at java.net.SocketOutputStream.socketWrite0(Native Method)

It looks to me like the buffers are full - server is not reading fast enough or the network is slow - you need to run a network monitor such as wireshark