Understanding JVM Memory Optimization

Question

I have recently taken up task to benchmark and optimize memory utilization in our Java Servers, while fixing the memory leaks if any. We have 4 JVM servers running in a single machine which has 15G Physical memory and 32G Swap memory. Following is free -m snapshot.

              total        used        free         shared  buff/cache    available
Mem:          15289       14786         392           1         110         342
Swap:         32767        3776       28991

If I understand correctly:-

1. Most of the physical memory is in use and most of the swap memory is free.
2. A low of amount of memory is actually buffered.
3. I have 48GB of virtual memory

Now each of my server is running with 7G Maximum heap size set with -Xmx option. Following is the output from GC Utils

$ jstat -gcutil 8317
  S0     S1     E      O      M     CCS      YGC     YGCT    FGC    FGCT     GCT
  0.00   0.00   6.97   0.22  96.77  91.89     72    8.027    33   10.975   19.002
$ jstat -gcutil 8332
  S0     S1     E      O      M     CCS      YGC     YGCT    FGC    FGCT     GCT
  0.00   0.00   5.17  50.76  96.57  91.65    274   51.001    51  179.106  230.107
$ jstat -gcutil 8249
  S0     S1     E      O      M     CCS      YGC     YGCT    FGC    FGCT     GCT
  0.00   2.10  80.37  69.53  96.87  92.08    421   69.478    13   56.569  126.047
$ jstat -gcutil 23875
  S0     S1     E      O      M     CCS      YGC     YGCT    FGC    FGCT     GCT
 27.91   0.00  37.07  86.29  97.59  94.60    232    7.294     2    0.030    7.324

After running manual Full GC, it completely freed young and old generation for all the servers. Now I need help in understanding this characterstic, basically am I correct in assuming following?

1. Objects move depending on longevity and surviving count as Eden -> S0 -> S1 -> O.
2. Three of the servers have Old Gen > Young Gen. Does that mean that the Full GC is not running enough?
3. Or, I need to tweak SurvivorRatio and NewRatio etc. settings to give Young Generation more chance to collect objects, and delay moving the objects to old space.
4. However increasing Y space means slower short GC and poor application throughput, Should I increase memory and stop worrying about tweaking?

I am asking this, because Full GC does free all memory, so there is no memory leak but How can I strike a balance between memory usage and application troughput by adjusting GC settings. Could anyone provide more direction or reccomendation by taking above machine state into consideration?

JVM Details:
Java Version: openjdk version "1.8.0_131" Server Class
GC: Parallel Collector(Default one)

Answer 1

In short: Basically going for memory usages at a certain point in time is a bad idea, if you want to understand your system. It is a lot more helpful to look at the behaviour over time. To do so, use wither the JConsole or professional profilers like YourKit or JProfiler. On the other hand, the defaults of the Hotspot-VM are quite reasonable, so unless you have a very specific application-memory-profile, you will probably not win a lot by fiddling and win more by simply throwing more hardware at the problem.

Be aware, that your server is most probably under-dimensioned (see question 4)

Long answer:

Question 1: How do objects proceed through the different memory-areas? Each object is created in eden space. The first minor garbage-collection (which is also triggered a major collections) will put that object either in survivor (if there is sufficient space) or directly into tenured. For objects already in survivor, they are moved from one survivor space to the other and at some point (determined by a heuristic within the VM) are moved into tenured. The rule always holds, that at least one survivor is always empty.

Question 2: Does a greater occupation of old than young mean, you are not collecting often enough? Definitely not. Garbage collection in Java works the better (more efficient), if it is executed seldom. Basically, if more objects died since the last collection, the collection is more efficient as it can clean up more memory at once. The gc is designed to let tenured to fill up until it reaches a very high fill level (close to 100%) prior to triggering a full-gc. We observe production systems not doing a full-gc for weeks because the young generations are large enough so that all temporary allocations die within eden or survivor. Apart from that: All your caches, Spring Beans or whatever will always be placed in tenured.

Question 3: Should I tweak memory parameters to optimize memory-consumption? By all means: NO! Never touch that stuff unless you know exactly, what you are doing. Enlarging the young generations might cause your application to die of out-of-memory-errors. You should be very clear about the long-term-behaviour of your application before fiddling with that stuff and the heuristics of the hotspot-VM are quite good. On the other hand, it is possible to significantly reduce gc-times if you know, what you are doing. In one system (the one, mentioned above) we saw a decrease of minor collections from once ~20 seconds to once in several minutes and an almost complete stop in full-gcs after tweaking. But in the process of learning and understanding, how our application behaved memory-wise over time, we also killed a lot executions.

Question 4: Should I increase memory instead of fiddling? Depends. Your system is not healthy as it is. Increasing memory will make that worse. You are running 4 VMs with a heap of up to 7g (total 28g) on a system with 16g physical memory. If all your VMs expand memory consumption to the maximum, you will see a lot of swapping and that will definitely reduce performance to a level you can hardly imagine. I would go for more hardware in the scenario as it is, especially considering, that the real consumption of a java application is roughly 1.5 times the heap (you have to add permgen/metaspace, the stack, codecache and the memory the vm itself consumes). If you know for sure, your application is running with -xmx7g you could start there: Increasing the overall memory while reducing the old-gen part so that the absolute old-size stays unchanged. That however means a lot of more hardware.