Troubleshooting Java memory usage exception

Question

I'm trying to troubleshoot a Java program that requires increasingly more memory until it cannot allocate any more and then it crashes.

EDIT More information about the program. The program is an indexer going through thousands of documents and indexing them for search. The documents are read from MongoDB and written to MongoDB as well after some processing is performed. During the processing I'm using RocksDB (rocksdb-jni version 5.13.4 from Maven). There is some mentioning in this GitHub issue of RocksDB memory usage growing uncontrollably, but I'm not sure it could be related.

Monitoring the process with visualvm results in the following plot:

but running htop on the machine shows totally different stats:

There is a difference of several GBs of memory that I'm unable to trace the source of.

The program is launched with the following VM arguments:

jvm_args: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=<port> -Djava.rmi.server.hostname=<hostname> -Xmx12G -XX:+UseStringDeduplication 

The system has 32GB of RAM and no swap. Out of this 32 GB, ~10GB are always taken by a tmpfs partition, ~8GB by MongoDB and the remaining 12GB are assigned to the program. EDIT The visualvm screenshot above shows 20GB of heap size, because it was from a previous run where I passed -Xmx20G; the behaviour, however, is the same whether I assign 12GB or 20GB to the heap. The behaviour also does not change if I remove the tmpfs partition, freeing 10 more GB of memory: it just takes longer but eventually it will get out of memory.

I have no idea where this memory usage that is not shown in visualvm but appears in htop is coming from. What tools should I use to understand what is going on? The application is running on a remote server, so I would like a tool that only works in the console or can be configured to work remotely, like visualvm.

Answer 1

I always use JProfiler but i hear jetbrains has a good one as well both can connect remotely.

If possible i would try to create a (local) setup where you can freely test it.

• in the RocksDB several possible solutions are mentioned, do they work?
• RocksDB seems to need some configuration, how did you configure it?
• I am not familiar with RocksDB but i see there is some indexing and caching going on. How much data are you processing and what indexes/caching configuration do you have? are you sure this should fit in memory?
• as far as i know the memory mismatch is because jni usage is not shown by default by most. There are some flags to improve this NativeMemoryTracking. i can't recall if this will add them to your visualvm overviews as well.

karbos-538 i just noticed this is quite an old issue, i hope this application is already working by now. what part of the question is still relevant to you?