CODEWITHSUNDEEP
javajava-11jemalloc
tuk
tuk

Reputation: 6872

Jemalloc: Java Native Memory profiling shows 100% je_prof_backtrace

We have a multi-threaded production Java application. We are trying to check the native memory usage as mentioned in this post.

But on the dump I am seeing 100% memory is being taken by je_prof_backtrace

ubuntu@platform1:/tmp/jemalloc_dump$ /home/ubuntu/jemalloc/bin/jeprof --show_bytes `which java` /tmp/jemalloc_dump/tsdb.62634.6454.i6454.heap
Using local file /usr/bin/java.
Using local file /tmp/jemalloc_dump/tsdb.62634.6454.i6454.heap
Welcome to jeprof!  For help, type 'help'.
(jeprof) top
Total: 815965512 B
815965512 100.0% 100.0% 815965512 100.0% je_prof_backtrace
       0   0.0% 100.0% 268451840  32.9% 0x00007f1338bd83a5
       0   0.0% 100.0% 279057735  34.2% 0x00007f13390d615a
       0   0.0% 100.0% 268455936  32.9% 0x00007f133918adda
       0   0.0% 100.0% 268455936  32.9% AllocateHeap
       0   0.0% 100.0% 268455936  32.9% JVM_MonitorWait
       0   0.0% 100.0% 268451840  32.9% Java_java_util_zip_Inflater_inflateBytesBytes
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::inflate
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::omAlloc
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::wait```

Output is like below if I use text

(jeprof) text
Total: 815965512 B
815965512 100.0% 100.0% 815965512 100.0% je_prof_backtrace
       0   0.0% 100.0% 268451840  32.9% 0x00007f1338bd83a5
       0   0.0% 100.0% 279057735  34.2% 0x00007f13390d615a
       0   0.0% 100.0% 268455936  32.9% 0x00007f133918adda
       0   0.0% 100.0% 268455936  32.9% AllocateHeap
       0   0.0% 100.0% 268455936  32.9% JVM_MonitorWait
       0   0.0% 100.0% 268451840  32.9% Java_java_util_zip_Inflater_inflateBytesBytes
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::inflate
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::omAlloc
       0   0.0% 100.0% 268455936  32.9% ObjectSynchronizer::wait

Steps followed.

  1. Downloaded jemalloc 5.2.1 release tag from here.
  2. ./autogen.sh --enable-prof inside the directory where jmalloc tar in untared.
  3. make

Application was started with the below options

export JEMALLOC_PATH=/home/ubuntu/jemalloc
export MALLOC_CONF=prof:true,lg_prof_interval:21,lg_prof_sample:28,prof_prefix:/tmp/jemalloc_dump/tsdb
LD_PRELOAD=${JEMALLOC_PATH}/lib/libjemalloc.so.2 \
java ...

I am new to jemalloc. Can someone suggest what could be going wrong?

ubuntu@platform1:/tmp/jemalloc_dump$ java -version
openjdk version "11.0.11" 2021-04-20 LTS
OpenJDK Runtime Environment Zulu11.48+22-SA (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.48+22-SA (build 11.0.11+9-LTS, mixed mode)

Upvotes: 0

Views: 1241

Answers (2)

tuk
tuk

Reputation: 6872

This has been discussed in github also. Posting the relevant reply

Quoting Jason from the original thread:

This is usually caused by the backtracing facility failing to do anything beyond reporting the caller's return instruction pointer. You may have better luck if you use libunwind, but unfortunately I've seen all the supported backtracing mechanisms fail under various conditions.

In other words, if you haven't tried libunwind, give that a try. The reported issue happens when the stack unwinder cannot walk the stack, in which case jemalloc / jeprof doesn't have any backtrace data to report.

Closing because nothing actionable from the jemalloc side.

Upvotes: 0

Pranjul Ahuja
Pranjul Ahuja

Reputation: 26

Try configuring using the below flags :

./configure --enable-prof --enable-stats --enable-debug --enable-fill

Upvotes: 1

Related Questions