Shutting down Netty server when client connections are open

Question

I'm trying to shutdown a Netty server that has open connections to it, and it just hangs. Here's what I do.

• Start server on one machine and client on another.
• Send a message from client to server to which I get a response.
• Shutdown server using Ctrl-C

I've registered a shutdown hook on the server that closes the ChannelGroup and calls releaseExternalResources on the ServerBootstrap (or actually I'm using the DuplexTcpServerBootstrap of the protobuf-pro-duplex library that does just that). Anyway, the shutdown hook is called properly on shutdown, but it never returns. When I take a thread dump of what's happening I can see two interesting stacks:

   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000006b0890950> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
    at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1433)
    at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:103)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.releaseExternalResources(AbstractNioWorkerPool.java:80)
    at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releaseExternalResources(NioServerSocketChannelFactory.java:162)
    at org.jboss.netty.bootstrap.Bootstrap.releaseExternalResources(Bootstrap.java:319)
    at com.googlecode.protobuf.pro.duplex.server.DuplexTcpServerBootstrap.releaseExternalResources(DuplexTcpServerBootstrap.java:132)
    at com.xxx.yyy.node.NodeServer$2.run(NodeServer.java:104)
    at java.lang.Thread.run(Thread.java:722)

So this is the shutdown hook thread that never returns. And below is another thread that seems to be waiting on a channel:

   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
    at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:274)
    at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
    - locked <0x00000006b0896660> (a java.lang.Object)
    at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
    at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:80)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    - locked <0x00000006b08964a8> (a sun.nio.ch.Util$2)
    - locked <0x00000006b0896498> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000006b0890d20> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:208)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

I'm using Netty 3.4.6.Final with Java 7.04 on Linux. Thanks!

Br, Frank.

Answer 1

I had same problem and solved it. You have to shutdown all EventLoopGroup synchronously and then close port.

It may take 4-5 seconds to completely shutdown.

Here is an example code (I think you should make a simple GUI with start and stop button to test it):

public class ReusePortServer {
    private final int port;
    ChannelFuture f;
    EventLoopGroup group;
    EpollEventLoopGroup bossGroup;
    EpollEventLoopGroup workerGroup;


    public ReusePortServer(int port) {
        this.port = port;
    }

    public void start() throws Exception {

        group = new EpollEventLoopGroup();

        bossGroup = new EpollEventLoopGroup();
        workerGroup = new EpollEventLoopGroup();

        ServerBootstrap b = new ServerBootstrap();
        b.group(bossGroup, workerGroup)
                .channel(EpollServerSocketChannel.class)
                .option(EpollChannelOption.SO_REUSEPORT, true)
                .childHandler(new ChannelInitializer<SocketChannel>() {
                    @Override
                    public void initChannel(SocketChannel ch) throws Exception {
                        ch.pipeline().addLast(new ReusePortHandler());
                    }
                });

        f = b.bind(port).sync();
        log(String.format("%s started and listen on %s", ReusePortServer.class.getName(), f.channel().localAddress()));
    }

    private final static SimpleDateFormat datefmt = new SimpleDateFormat("HH:mm:ss ");

    public static void log(final String msg) {
        System.out.print(datefmt.format(new Date()));
        System.out.println(msg);
        System.out.flush();
    }

    public void stop() {
        System.out.println("ReusePortServer.stop");
        try {
            // shutdown EventLoopGroup
            bossGroup.shutdownGracefully().sync();
            workerGroup.shutdownGracefully().sync();
            f.channel().closeFuture().sync();   // close port

        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    public static void main(final String[] args) throws Exception {
        int port = 12345;
        new ReusePortServer(port).start();
    }
}

Note: You can change EpollEventLoopGroup to any EventLoopGroup you want.

Answer 2

Had the same 'problem' with bare Netty client/server as well.

Thing is, closing the server channel DOES NOT close the open channels created for accepted client connections. One has to explicitly keep track of client channels in the server. This can be done with channel groups and a handler which adds client channels to this group. When it comes to shutting down the server, all channels in the group should be closed in a batch-like manner instead of just the one server channel (which can put into the channel group as well).

There is excellent documentation in the user guide (9. Shutting Down Your Application): http://static.netty.io/3.5/guide/ and the ChannelGroup API doc (Simplify shutdown process with ChannelGroup): http://static.netty.io/3.5/api/org/jboss/netty/channel/group/ChannelGroup.html

Answer 3

Netty Server Shutdown

1. Close server channel
2. Shutdown boss and worker executor
3. Release server bootstrap resource

Example code

ChannelFuture cf = serverChannel.close();
cf.awaitUninterruptibly();
bossExecutor.shutdown();
workerExecutor.shutdown();
thriftServer.releaseExternalResources();