WebSphere out of memory error

Question

We use WebSphere application server for our application and we regularly get out of memory error. To debug this we added log to check used memory at certain places and below is the observation.

The used memory is not decreasing until it reaches threshold limit. We use below memory configuration:

InitialHeapSize="1024" maximumHeapSize="2048"

So until it crosses 1024 the memory is not released. In the case of OOM error, the memory is not released only even though some threads are not in use.

I assumed that the heap size was not released. But the java Runtime API is displaying that there is memory available. Java operations like method class, string opertaions are working but its failing when JNDI look up is made with outofmemory exception. As a result, the system is failing because of unavilability of connection.

Stack trace:

com.ibm.websphere.naming.CannotInstantiateObjectException: Exception occurred while the JNDI NamingManager was processing a javax.naming.Reference object. [Root exception is java.lang.OutOfMemoryError] at com.ibm.ws.naming.util.Helpers.processSerializedObjectForLookupExt(Helpers.java:1033) at com.ibm.ws.naming.util.Helpers.processSerializedObjectForLookup(Helpers.java:730)

Answer 1

There will be some differences depending on the platform and edition you are using, but, there is built in support for generating heap dumps:

See, for example:

http://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tprf_enablingheapdump.html

Generally, you will either want to enable generation of heap dumps, then force an OOM, then use the HeapAnalyzer to analyze the resulting heap dump. Or, you can manually generate heaps when large memory usage is seen.

Some caution: What may look like a memory leak may be a very large but transient memory use. A view of memory usage over time will be needed to conclude that there is an actual leak.

Regardless, the path for handling this sort of problem inevitably leads to generating a heap dump and doing analysis.

Answer 2

You need to generate Heap Dump and Thread Dump via wasadmin and analyze for root causes

Answer 3

Dynamo , you will have to perform a heap analysis to find out what causes the OOM for your. It is a free tooling that allows you to find out what is causing the issue in the server. May be it is a rogue application that is blocking too much memory or a resource that is leaking too much memory etc.

you can look at this for more information. Your setting of initial heap and maximum heap is something you want to tune (If you have it too deep for GC , your CPU will be very high during GC vs constant overhead usage issues if it is too frequent)

https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=4544bafe-c7a2-455f-9d43-eb866ea60091