CODEWITHSUNDEEP
javagarbage-collectionjvmmonitoringheap-memory
cruftex
cruftex

Reputation: 5723

How to detect a low heap situation for monitoring and alerting purposes?

We monitor our production JVMs and have monitoring triggers that (ideally) send warnings, when the JVM runs low on heap space. However, coming up with an effective detection algorithm is quite difficult, since it is the nature of garbage collection, that the application regularly has no available memory, before the GC kicks in.

There are many ways to work around this, I can think of. E.g. monitor the available space, send a warning when it becomes too low, but delay it and only trigger, when it is persistent for more than a minute. So, what works for you in practice?

Particular interesting:

Upvotes: 2

Views: 1560

Answers (1)

Jared
Jared

Reputation: 26149

I have found a very effective measure of JVM memory health to be the percentage of time the JVM spends in garbage collection. A healthy, well-tuned JVM will use very little (< 1% or so) of its CPU time collecting garbage. An unhealthy JVM will be "wasting" much of its time keeping the heap clean, and the percent of CPU used on collection will climb exponentially in a JVM experiencing a memory leak or with a max heap setting that is too low (as more CPU is used keeping the heap clean, less is used doing "real work"...assuming the inbound request rate doesn't slow down, it's easy to fall off a cliff where you become CPU bound and can't get enough work done quickly enough long before you actually get a java.lang.OutOfMemoryError).

It's worth noting that this is really the condition you want to guard against, too. You don't actually care if the JVM uses all of its heap, so long as it can efficiently reclaim memory without getting in the way of the "real work" it needs to do. (In fact, if you're never hitting the max heap size, you may want to consider shrinking your heap.)

This statistic is provided by many modern JVMs (certainly Oracle's and IBMs, at least).

Another somewhat effective measure can be the time between full GCs. The more often you are having to perform a full GC, the more time you're spending in GC.

Upvotes: 2

Related Questions