CODEWITHSUNDEEP
nettygrpc-javanetty4
user2649857
user2649857

Reputation: 21

Using Netty Server for gRPC results in Out of Memory Error

Using Netty Server Builder for gRPC Server & running into Out of Memory error where direct memory size is exceeding the allocated size of 1.2g.

Tried using both pooled byte buffer allocator as well as unpooled byte buffer allocator. Looks like Netty is creating Thread Local Cache and the size of the thread local cache is almost equal to the total size of direct memory. Tried setting the below system property to disable thread local cache but still see that the thread local cache is getting created.

Initially was using Netty version 4.1.74.FINAL but also upgraded Netty server version 4.1.89.Final and seeing the same issue

Any suggestions on how to use Netty Server effectively for creating & managing the gRPC server.

Tried using both pooled byte buffer allocator as well as unpooled byte buffer allocator. Looks like Netty is creating Thread Local Cache and the size of the thread local cache is almost equal to the total size of direct memory. Tried setting the below system property to disable thread local cache but still see that the thread local cache is getting created.

Upvotes: 1

Views: 1812

Answers (1)

Larry Safran
Larry Safran

Reputation: 177

I would suggest that you either use upooled or a small thread local cache:

  1. set io.netty.allocator.type="unpooled"
  2. set io.netty.threadLocalDirectBufferSize=16384

This is based on the default for threadLocalDirectBufferSize resulting in threadlocal buffers large enough to hold the entire message (which since you said that each are using up almost all of the memory must be large) and that the system is going to break large messages up into 16KB chunks for sending over the wire, so you are just moving where the chunking is occurring.

Original that generated comments:

You didn't list the command that you had tried. Was it

-Dio.netty.allocator.useCacheForAllThreads=false

That should only be relevant with PooledByteBufferAllocator.

If you are using UnpooledByteBufAllocator, then there shouldn't be any thread local caches.

I am assuming you are following standard practice and setting the allocator in ChannelConfig, is that correct?

Upvotes: 0

Related Questions