Major GC decreasing performance

Question

We are having frecuent outages in our app, basically the heap grows over time to the point were the GC takes a lot of CPU time and execute for several minutes, decreasing the app perfomance drastically. The app is in JSF with a tomcat server.

In the mean time, we:

• Increased the heap size from 15G to 26G (-Xms27917287424 -Xmx27917287424)
• Take several heap dumps (we are trying to determine the problem using these)
• Activated GC logs

With the heap size increase, GC is not executing for that much time but still takes a lot of CPU and frezees the app.

So the question is:

• Is this normal? When the GC executes it frees memory, so i think this probably isn't a memory leak (Am I right?)
• Is there a way of optimize the GC or maybe this behavior is just a sympthom of something wrong in the app itself?
• How can I monitor and analyze this without taking a heap dump?

UPDATE: I changed JSF from 2.2 to 2.3 because some heap dumps were pointing that JSF was using a lot of memory. That didn't work out, and yesterday we had and outage again, but this time a little different (from my point of view). Also this time, we had to reset tomcat because the app didn't work anymore after a while

In this case, the garbage collector is running when de old gen heap is not full, and the new generation GC is running all the time. ¿What can be the cause of this?

Answer 1

As has been said in the comments, the behaviour of the application does not look unreasonable. Your code is continually allocating objects that leads to heap space filling up, causing the GC to run. There does not appear to be a memory leak since GC reclaims a lot of space and the overall used space is not continually increasing.

What does appear to be an issue is that a significant number of objects are being promoted to the old-gen before being collected. Major GC cycles are more expensive in terms of CPU due to the relocation and remapping of objects (assuming you're using a compacting algorithm).

To reduce this, you could try increasing the size of the young generation. This will have happened when you increased the overall heap size but not by enough. Ideally, you want the majority of objects to be collected during a minor GC cycle since this is effectively free (the GC does nothing to the objects in Eden space as they are collected). You can do this with the -XX:NewRatio= or -XX:NewSize= flags. You could also try changing the survivor space sizes, again to increase the number of objects collected before tenuring. (use the -XX:SurvivorRatio= flag for this).

For monitoring, I find Flight Recorder and Mission Control very useful as you can drill down into details of how many objects of specific types are allocated. It's also easy to connect to a running JVM or take dumps for later analysis.