CODEWITHSUNDEEP
javadockerkubernetesmemory-leaksgrafana
Albert Hendriks
Albert Hendriks

Reputation: 2142

Java runs out of memory. Is this not a memory leak?

I have assigned our Java program 2GB of memory. During the hours a particular thread is running, memory steadily and linearly increases until Kubernetes kills it because it reaches the 2GB limit I assigned. Of course we were thinking of a memory leak, but we see something like this all the time in the gc log:

[7406.381s][info][gc] GC(8326) Pause Full (System.gc()) 130M->65M(214M) 157.995ms
  1. Since the memory increases linearly while these logs indicates that the heap memory does not increase, is it useless to investigate memory leaks?
  2. What could be other likely causes of the increasing memory?

Some background info:

There are no logs that say the container was stopped or killed. There are also no events in k8s (however "restarts" = 1). The above log line was the last log line before we see (in Graylog) that Spring Boot / Tomcat is starting (hence it must have been restarted). We see this happening exactly at the time when the memory graph reaches the 2GB line in Grafana. Without Grafana it would have taken a while before we figured out it was something related to memory.

Kubernetes deploy yml part:

spec:
  template:
    spec:
      containers:
        - name: ... (omitted)
          resources:
            limits:
              cpu: 1200m
              memory: 2Gi
            requests:
              cpu: 50m
              memory: 50Mi

Last line of Dockerfile:

ENTRYPOINT ["java", "-Xmx2G", "-verbose:gc", "-jar", "/backend.jar"]

where "-verbose:gc" causes the log lines like the line I quoted above.

It takes a while to reproduce the problem, but we did that a couple of times.

We're using Java 11.

Upvotes: 1

Views: 2171

Answers (1)

ewramner
ewramner

Reputation: 6278

I don't think you have a leak at all, you are just using the options wrong. With -Xmx2G you are telling Java that it can use up to 2G for the heap. At the same time you are telling Kubernetes that the absolute limit for memory is 2Gi. Now, Java uses memory that is not on the heap, so when it tries to expand the heap to 2G it runs out and the pod is killed.

To fix the problem make sure that you allow a reasonable margin for the memory that is outside the heap. Increase the Kubernetes limit to 3G temporarily and then scale it down when you know how much native memory you need. I would guess that 2.5G is a reasonable level, but that is just a guess. Alternatively you can decrease the Java heap size and run with a 1.5G heap (or less) to leave some room for the native memory.

Upvotes: 5

Related Questions