Why a netty client performs as zombie?

Question

I use netty as socket client:

public void run() {
    isRunning = true;
    EventLoopGroup group = new NioEventLoopGroup(EventLoopsPerGetter);
    Bootstrap b = new Bootstrap();
    b.group(group).channel(NioSocketChannel.class)
     .handler(new ChannelInitializer<SocketChannel>() {

        @Override
        protected void initChannel(SocketChannel ch) throws Exception {
            ChannelPipeline p = ch.pipeline();
            p.addLast(
                    new ProtobufVarint32FrameDecoder(),
                    ZlibCodecFactory.newZlibDecoder(ZlibWrapper.GZIP),
                    new ProtobufDecoder(Protocol.Packet.getDefaultInstance()),

                    new ProtobufVarint32LengthFieldPrepender(),
                    ZlibCodecFactory.newZlibEncoder(ZlibWrapper.GZIP),
                    new ProtobufEncoder(),

                    session
                    );
        }
    });
    try {
        while(isRunning) {
            try {
                 b.connect(host, port).sync().channel().closeFuture().sync();
            } catch(Exception e) {
                if (e instanceof InterruptedException) {
                    throw e;
                }
                retryLogger.warn("try to connect to " + host + " : " + port + " , but", e);
            }
            if(isRunning) {
                retryLogger.info("netty connection lost, retry!");
                Thread.sleep(RetryInterval);
            }
        }
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
    } finally {
        group.shutdownGracefully();
    }
}

The session code is very simple, send Get-packet to server, get response, write file, then send next Get-packet.

In this program, I start two netty client threads, but after running several days, one of them behaves like a zombie thread, that is, even if I kill the netty server, the zombie client prints no log while the other client prints the wanted logs. By the way, the jstack file shows both threads are live, not dead.

I am using netty 5.

Answer 1

You don't have any mechanism for a readtimeout, what happens is that there is no traffic for 10~ (depends on the router model) minutes and the NAT table in the router thinks the connection is done, and closes the connection.

You have multiple ways to solve this problem:

Using ReadTimeoutHandler

ReadTimeoutHandler closes the channel and throws a ReadTimeoutException if a timeout is detected. You can catch this exception if needed via the exceptionCaught. With your existing logic, you don't need to catch this.

This handler can also be used in combination with a WriteTimeoutHandler to write "ping" messages to the remote. However the following solution is better for this purpose.

Using IdleStateHandler

You can also use a IdleStateHandler for this purpose, this handler has 3 arguments that stand for readerIdleTime, writeIdleTime and allIdleTime. The advantage of this class is that it doesn't throw exceptions and uses the Netty userEventTriggered to dispatch its calls, while this makes the class harder to use, you can do more things with it.

For example, if you protocol supports ping messages, you can use this class to send to send those ping messages. Its really easy to use this class and can be used in handlers as the following:

public class MyChannelInitializer extends ChannelInitializer<Channel> {
     @Override
     public void initChannel(Channel channel) {
         channel.pipeline().addLast("idleStateHandler", new IdleStateHandler(60, 30, 0));
         channel.pipeline().addLast("myHandler", new MyHandler());
     }
 }

 // Handler should handle the IdleStateEvent triggered by IdleStateHandler.
 public class MyHandler extends ChannelHandlerAdapter {
     @Override
     public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
         if (evt instanceof IdleStateEvent) {
             IdleStateEvent e = (IdleStateEvent) evt;
             if (e.state() == IdleState.READER_IDLE) {
                 ctx.close();
             } else if (e.state() == IdleState.WRITER_IDLE) {
                 ctx.writeAndFlush(new PingMessage());
             }
         }
     }
 }