How to find a root cause of the following netty error: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s)

Question

I've experienced the following exception thrown from the pipeline responsible for sending data to a TCP client.

2017-03-02T18:00:53,749 [epollEventLoopGroup-3-1] ERROR [ExceptionHandler - null] - Unknown exception
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 117440512, max: 119537664)
    at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:613) ~[netty-common-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:567) ~[netty-common-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:699) ~[netty-buffer-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:688) ~[netty-buffer-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237) ~[netty-buffer-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:213) ~[netty-buffer-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:141) ~[netty-buffer-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:262) ~[netty-buffer-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170) ~[netty-buffer-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:131) ~[netty-buffer-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:73) ~[netty-transport-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.channel.RecvByteBufAllocator$DelegatingHandle.allocate(RecvByteBufAllocator.java:124) ~[netty-transport-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:950) [netty-transport-native-epoll-4.1.2.Final-linux-x86_64.jar:4.1.2.Final]
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:359) [netty-transport-native-epoll-4.1.2.Final-linux-x86_64.jar:4.1.2.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:400) [netty-common-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:306) [netty-transport-native-epoll-4.1.2.Final-linux-x86_64.jar:4.1.2.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:805) [netty-common-4.1.2.Final.jar:4.1.2.Final]
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145) [netty-common-4.1.2.Final.jar:4.1.2.Final]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60] 

The component that reports this bug sends data in chunks. The chunk size differs from the range of 512b to 15kb

Could you please advice me regarding the best tools and techniques to find the root cause of this error.

Answer 1

Above exception could have 2 reasons:

1. You have memory leak in your app (most probably);
2. You really create big load or allocate many objects in direct memory, so you need more direct memory;

For case 1 you need to enable paranoid level of objects tracking in order to detect possible memory leak with -Dio.netty.leakDetection.level=paranoid property during your server start (or -Dio.netty.leakDetection.level=advanced level at least). With this option enabled netty will notify you with log messages in case memory leak is detected so you'll be able to fix your pipeline. You need to start from this point. More info.

If above not a case for you then - you may try to increase memory size with -XX:MaxDirectMemorySize= option as stated in above answer. Another option would be to turn off direct memory allocator with -Dio.netty.noPreferDirect=true. And last option - you can increase the memory allocated for your process. Seems like you have only 512 of RAM for your server.

One more recommendation would be to update to latest netty version. As some previous versions had memory leaks in netty itself.

Answer 2

In your case, the direct memory size has been exhausted. According to the code from Class io.netty.util.internal.PlatformDependent, where you can find some comments here:

    // Here is how the system property is used:
    //
    // * <  0  - Don't use cleaner, and inherit max direct memory from java. In this case the
    //           "practical max direct memory" would be 2 * max memory as defined by the JDK.
    // * == 0  - Use cleaner, Netty will not enforce max memory, and instead will defer to JDK.
    // * >  0  - Don't use cleaner. This will limit Netty's total direct memory
    //           (note: that JDK's direct memory limit is independent of this).
    long maxDirectMemory = SystemPropertyUtil.getLong("io.netty.maxDirectMemory", -1);

The suggestion is modify the JVM runtime parameter by command lines with ' -XX:MaxDirectMemorySize='

---

I was confused of your description: 'The chunk size differs from the range of 512b to 15kb'. Well, the log suggests another size 'failed to allocate 16777216 byte(s)' 16Mbytes. Maybe you need to check your application for the package size.

---

Usually, I modify some bootstrap options when using Netty, like

            bootstrap.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 3000)
                .option(ChannelOption.SO_SNDBUF, 1024 * 256)
                .option(ChannelOption.SO_RCVBUF, 1024 * 32768)
                .option(ChannelOption.TCP_NODELAY, true);

or

bootstrap.group(bossGroup, workerGroup)
            .option(ChannelOption.TCP_NODELAY, true)
            .option(ChannelOption.RCVBUF_ALLOCATOR, new FixedRecvByteBufAllocator(4096))
            .option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)