CODEWITHSUNDEEP
garbage-collectionjvmheap-dump
px5x2
px5x2

Reputation: 1565

JVM GC behaviour on heap dump and unnecessary heap usage

We have problem tuning the memory management of JVM's. The very same application running on the k8s cluster, but one of the pods' jvm heap usage rises to ~95% and, when we try to get a heapdump on this vs, somehow gc runs and heap usage drops suddenly, leaving us with a tiny heap dump.

I think the old space has grown unnecessarily, and gc did not work to reclaim memory (for nearly 15 hours). Unfortunately we can't see what is occupying the space, because the heap dump is very small as gc is forced.

All 3 pods are having memory of 1500m and here is the jvm heap usage percentage graph (3 pods, green being the problematic one):

JVM Heap Usage

Details:

openjdk 15.0.1 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)

JVM Parameters:

-XX:MaxRAMPercentage=75
-XX:InitialRAMPercentage=75
-server
-Xshare:off
-XX:MaxMetaspaceSize=256m
-Dsun.net.inetaddr.ttl=60
-XX:-OmitStackTraceInFastThrow
-XX:+ShowCodeDetailsInExceptionMessages

The questions are:

Upvotes: 0

Views: 1583

Answers (1)

Alexey Ragozin
Alexey Ragozin

Reputation: 8409

JVM heap dump procedure has 2 modes

  • live objects - this mode executes Full GC along side with heap dump. This is default options.
  • all objects - heap dump would include all object on heap both reachable and unreachable.

Heap dump mode is usually possible to choose via tool specific option.

Answering your questions

Why a full gc is called when we try to get heap dump?

Answered above

What is the motivation behind the gc not reclaiming memory and causes the application run with the heap size between ~70% and ~95%, while jvm can use and perfectly work with only 10%?

Reclaiming memory required CPU resources and impacts application latency. While JVM is operating withing memory limits it will mostly avoid expensive GC.

Recent development of containers is driving some changes in JVM GC department, but statement above is still relevant for default GC configuration.

What can be done to force jvm to do gc more aggressively to avoid this situation? Or should it be done for production environment?

Original answers lack problem statement. But general advises are

  • manage memory limits per container (JVM derive heap size from container limits unless they are overridden explicitly)
  • forcing GC periodically is possible, though unlikely to be a solution to any problem
  • G1GC has wide range of tuning options relevant for containers

Upvotes: 0

Related Questions