CODEWITHSUNDEEP
javamemoryherokujvm
Edi
Edi

Reputation: 640

Heroku JVM memory quota exceeded but the numbers don't add up

We have a JVM app running on Heroku which is receiving the Error R14 Memory quota exceeded.

We're trying to understand where the memory is being spend, but the numbers don't add up.

Below the logs:

 Process running mem=1174M(114.7%)
 » 10:16:28.365 2015-01-21 09:16:28.099931+00:00 heroku web.1 - - Error   R14 (Memory quota exceeded) Critical
 » 10:16:43.322 2015-01-21 09:16:42.836517+00:00 app web.1 - - measure.mem.jvm.heap.used=272M measure.mem.jvm.heap.committed=546M measure.mem.jvm.heap.max=546M
 » 10:16:43.322 2015-01-21 09:16:42.836583+00:00 app web.1 - - measure.mem.jvm.nonheap.used=106M measure.mem.jvm.nonheap.committed=107M measure.mem.jvm.nonheap.max=0M
 » 10:16:43.322 2015-01-21 09:16:42.836644+00:00 app web.1 - - measure.threads.jvm.total=136 measure.threads.jvm.daemon=21 measure.threads.jvm.nondaemon=105 measure.threads.jvm.internal=10
 » 10:16:43.322 2015-01-21 09:16:42.853114+00:00 app web.1 - - measure.mem.linux.vsz=2489M measure.mem.linux.rss=848M

Heroku reports 1174M being used. The heroku-javaagent-1.4 reports the metrics right under that, which add up to 546+107+136/2=721M. Where could the rest of 1174-721=453M be spent? How could we continue troubleshooting?

I've factored here 136 threads with 512K stack each, given our JVM options:

-javaagent:heroku-javaagent-1.4.jar=stdout=true,lxmem=true -Xms568m -Xmx568m -Xmn192m -Xss512k -XX:+UseCompressedOops

for running on a 2x dyno of 1024M memory.

Thanks

Upvotes: 1

Views: 1489

Answers (2)

ntepp jean marc
ntepp jean marc

Reputation: 108

It is true that this question has been around for a long time. But I met the same issue recently.

The solution was to configure a server for my static files. You can use Nginx for the static file.

Indeed, the loading of static files consumes memory by I / O requests.

Upvotes: 0

codefinger
codefinger

Reputation: 10318

There are a few possibilities here. The most likely is that something is allocating native memory maps for some kind of IO. Tools that report JVM memory usage can't see this. Most commonly, this happens by Java NIO ByteBuffer objects. This could be done by something in your code, which might indicate a memory leak in your app. Or it could be done by the framework/server your app is built on.

I've found that Play framework and Jetty are particularly bad about this. In many cases, they allocate hundreds of megabytes of native memory. Are you using either of these? I know people who have switched from Jetty to Tomcat and resolved the issue.

Ultimately, if the extra memory is reaching a steady state, it may be ok. It's just what your app needs to run. But if you find that keeps growing and growing, then it may indicate a memory leak.

I suggest filing a ticket with Heroku's support, and they can use smaps to provide a little more info on where this memory is being allocation.

Full disclosure: I work for Heroku.

Upvotes: 4

Related Questions