Vijaya
Vijaya

Reputation: 167

No buffer space available (maximum connections reached?) issue when triggering external Rest API using Camel and netty-http

I am using Camel and netty-http to consume the external rest api's of azure.

I have configured camel-cron to run my api 3 hours once, which will trigger azure api to pull all the builds and then for each of the build it will trigger subsequent azure rest api to get further details.

I have implemented this on top of Spring boot and deployed into Tomcat server.

It works well for 2 days then the below error is getting occur while triggering the external rest-api,

Reason: java.lang.IllegalStateException: failed to create a child event loop
    at org.apache.camel.support.cache.DefaultProducerCache.acquireProducer(DefaultProducerCache.java:155)
    at org.apache.camel.support.cache.DefaultProducerCache.doInAsyncProducer(DefaultProducerCache.java:283)
    at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:182)
    at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:388)
    at org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:107)
    at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:181)
    at org.apache.camel.impl.engine.DefaultReactiveExecutor.schedule(DefaultReactiveExecutor.java:57)
    at org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask.done(CamelInternalProcessor.java:264)
    at org.apache.camel.support.cache.DefaultProducerCache.lambda$doInAsyncProducer$4(DefaultProducerCache.java:329)
    at org.apache.camel.processor.SendDynamicProcessor$1.done(SendDynamicProcessor.java:213)
    at org.apache.camel.component.netty.http.NettyHttpProducer$NettyHttpProducerCallback.done(NettyHttpProducer.java:175)
    at org.apache.camel.component.netty.NettyProducer$NettyProducerCallback.done(NettyProducer.java:571)
    at org.apache.camel.component.netty.NettyCamelState.callbackDoneOnce(NettyCamelState.java:55)
    at org.apache.camel.component.netty.handlers.ClientChannelHandler.channelRead0(ClientChannelHandler.java:223)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1533)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: failed to create a child event loop
    at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
    at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
    at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
    at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
    at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86)
    at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81)
    at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68)
    at org.apache.camel.component.netty.NettyWorkerPoolBuilder.build(NettyWorkerPoolBuilder.java:79)
    at org.apache.camel.component.netty.NettyProducer.doStart(NettyProducer.java:137)
    at org.apache.camel.support.service.BaseService.start(BaseService.java:115)
    at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:113)
    at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:130)
    at org.apache.camel.component.rest.RestProducer.doStart(RestProducer.java:261)
    at org.apache.camel.support.service.BaseService.start(BaseService.java:115)
    at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:113)
    at org.apache.camel.impl.engine.AbstractCamelContext.internalAddService(AbstractCamelContext.java:1476)
    at org.apache.camel.impl.engine.AbstractCamelContext.addService(AbstractCamelContext.java:1394)
    at org.apache.camel.support.cache.ServicePool$SinglePool.acquire(ServicePool.java:210)
    at org.apache.camel.support.cache.ServicePool$SinglePool.acquire(ServicePool.java:193)
    at org.apache.camel.support.cache.ServicePool.acquire(ServicePool.java:109)
    at org.apache.camel.support.cache.DefaultProducerCache.acquireProducer(DefaultProducerCache.java:126)
    ... 53 more
Caused by: io.netty.channel.ChannelException: failed to open a new selector
    at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175)
    at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142)
    at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146)
    at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
    at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
    ... 73 more
Caused by: java.io.IOException: Unable to establish loopback connection
    at java.base/sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:94)
    at java.base/sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:61)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/sun.nio.ch.PipeImpl.<init>(PipeImpl.java:171)
    at java.base/sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
    at java.base/java.nio.channels.Pipe.open(Pipe.java:155)
    at java.base/sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:142)
    at java.base/sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
    at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173)
    ... 77 more
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
    at java.base/sun.nio.ch.Net.connect0(Native Method)
    at java.base/sun.nio.ch.Net.connect(Net.java:483)
    at java.base/sun.nio.ch.Net.connect(Net.java:472)
    at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:694)
    at java.base/java.nio.channels.SocketChannel.open(SocketChannel.java:194)
    at java.base/sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(PipeImpl.java:127)
    at java.base/sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:76)
    ... 85 more

Later, I have added below code to close the connection after use

.process(exchange -> exchange.getMessage().setHeader(NettyConstants.NETTY_CLOSE_CHANNEL_WHEN_COMPLETE, true))

But still issue exists.

Since the above exception occurs only while triggering the external rest-api, Wanted to check on below 2 points,

  1. Will Camel's netty-http closes the connection after triggering external rest-api?
  2. While triggering the external rest-api, will it open new port every time?

Can someone please help on this. Thanks in advance.

Upvotes: 0

Views: 263

Answers (0)

Related Questions