CODEWITHSUNDEEP
javamemorydockermemory-leaks
Knight71
Knight71

Reputation: 2949

How to find non-heap space memory leak in java?

We have a java webserver that is using eclipse-jetty version 8.1.6. Recently we started noticing Out of memory error. We had few profiling on the number of threads active. This seems to be reasonable around 100. The process has 5GB max heap memory and 4GB initial heap memory. 

Process Details
Environment: Docker(kubernetes)
java.version="1.8.0_91"
java.vm.info="mixed mode"
java.vm.name="Java HotSpot(TM) 64-Bit Server VM"
thread size = 1024K
ulimit is unlimited for max process per user.
Container(Pod) Max memory is allocated to be 8GB

The webserver receives on a average of 350 request per minute. Also we run many such instances behind ELB(kubernetes service). After running for few hours we notice this OOM. The issue is random and it occurs on stress test.

OOM StackTrace:

java.lang.OutOfMemoryError: unable to create new native thread
   at java.lang.Thread.start0(Native Method) [na:1.8.0_91]
   at java.lang.Thread.start(Thread.java:714) [na:1.8.0_91]
   at org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPool.java:441) [jetty-util-8.1.16.v20140903.jar:8.1.16.v20140903]
   at org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.java:366) [jetty-util-8.1.16.v20140903.jar:8.1.16.v20140903]

Since the number of thread count is reasonable. I suspect some memory leak. But I am not able to find the off heap memory size on a docker container. Is there a way to find it ?

After searching for a while ,I found this bug below in jetty.

How to verify that if the bug is due to below issue without upgrading jetty ?

Related Bug Id: https://bugs.eclipse.org/bugs/show_bug.cgi?id=477817

Upvotes: 2

Views: 1295

Answers (1)

Dmitry K
Dmitry K

Reputation: 184

You could try to add -XX:-HeapDumpOnOutOfMemoryError into your java start parameters and to look into the dump.

Upvotes: 1

Related Questions