How to fix Out Of Memory issue on sonarqube server 6.7.2.37468

Question

When Sonarqube server version 6.7.2 is running on AWS EC2 instance, once by month approximatively, there is an out of memory error and the server is shutdown, we need to restart it manually, there is a memory leak or bad setup configuration on OS/JRE level or Application server level.

At the beginning, the sonarqube server goes down twice a month due to Out Of Memory issue, we follow the official setup doc for PRD instance and try to scale vertically the capacity from a t2.small to a t2.medium on the EC2 instance and run the 3 java processes with more memory:

For Web:
 sonar.web.javaOpts=-Xms256m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError  

For ElasticSearch: 
 sonar.search.javaOpts=-Xms1024m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError

For Compute Engine:
 sonar.ce.javaOpts=-Xms256m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError

But the issue is still present

--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

2019.04.29 06:59:38 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating 
temp directory /opt/sonarqube-6.7.2/temp
2019.04.29 06:59:38 INFO  app[][o.s.a.es.EsSettings] Elasticsearch 
listening on /127.0.0.1:9001
2019.04.29 06:59:38 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch 
process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from                 [/opt/sonarqube-6.7.2/elasticsearch]: /opt/sonarqube-    6.7.2/elasticsearch/bin/elasticsearch -Epath.conf=/opt/sonarqube-    6.7.2/temp/conf/es
2019.04.29 06:59:38 INFO  app[][o.s.a.SchedulerImpl] Waiting for     Elasticsearch to be up and running
2019.04.29 06:59:39 INFO  app[][o.e.p.PluginsService] no modules loaded
2019.04.29 06:59:39 INFO  app[][o.e.p.PluginsService] loaded plugin     [org.elasticsearch.transport.Netty4Plugin]
2019.04.29 07:00:00 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2019.04.29 07:00:00 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch     process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube-    6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-    1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -    Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xmx1024m -Xms256m -    XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= -    Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost=     -Dhttps.proxyPort= -cp ./lib/common/*:./lib/server/*:/opt/sonarqube-    6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.server.app.WebServer     /opt/sonarqube-6.7.2/temp/sq-process5028201971842772853properties
2019.04.29 07:00:25 INFO  app[][o.s.a.SchedulerImpl] Process[web] is up
2019.04.29 07:00:25 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch     process[[key='ce', ipcIndex=3, logFilenamePrefix=ce]] from [/opt/sonarqube-    6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-    1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -    Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xms256m -Xmx1024m -    XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= -    Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost=     -Dhttps.proxyPort= -cp     ./lib/common/*:./lib/server/*:./lib/ce/*:/opt/sonarqube-    6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.ce.app.CeServer     /opt/sonarqube-6.7.2/temp/sq-process7912202285523412339properties
2019.04.29 07:00:30 INFO  app[][o.s.a.SchedulerImpl] Process[ce] is up
2019.04.29 07:00:30 INFO  app[][o.s.a.SchedulerImpl] SonarQube is up
2019.04.29 15:10:28 INFO  app[][o.s.a.SchedulerImpl] Process [ce] is     stopped
2019.04.29 15:10:31 INFO  app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.04.29 15:10:31 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
2019.04.29 15:10:31 INFO  app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.04.29 15:10:31 INFO  app[][o.s.a.SchedulerImpl] SonarQube is restarting
2019.04.29 15:10:31 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
2019.04.29 15:10:31 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube-6.7.2/temp
2019.04.29 15:10:31 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.04.29 15:10:31 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch     process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube-    6.7.2/elasticsearch]: /opt/sonarqube-6.7.2/elasticsearch/bin/elasticsearch -    Epath.conf=/opt/sonarqube-6.7.2/temp/conf/es
2019.04.29 15:10:31 INFO  app[][o.e.p.PluginsService] no modules loaded
2019.04.29 15:10:31 INFO  app[][o.e.p.PluginsService] loaded plugin     [org.elasticsearch.transport.Netty4Plugin]
2019.04.29 15:10:51 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2019.04.29 15:10:51 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch     process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube-    6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-        1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -        Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xmx1024m -Xms256m -    XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= -    Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost=     -Dhttps.proxyPort= -cp ./lib/common/*:./lib/server/*:/opt/sonarqube-    6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.server.app.WebServer     /opt/sonarqube-6.7.2/temp/sq-process2158511348006070798properties
2019.04.29 15:11:12 INFO  app[][o.s.a.SchedulerImpl] Process[web] is up
2019.04.29 15:11:12 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch     process[[key='ce', ipcIndex=3, logFilenamePrefix=ce]] from [/opt/sonarqube-    6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-    1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -    Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xms256m -Xmx1024m -    XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= -    Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost=     -Dhttps.proxyPort= -cp     ./lib/common/*:./lib/server/*:./lib/ce/*:/opt/sonarqube-    6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.ce.app.CeServer     /opt/sonarqube-6.7.2/temp/sq-process1917579452884722345properties
2019.04.29 15:11:17 INFO  app[][o.s.a.SchedulerImpl] Process[ce] is up
2019.04.29 15:11:17 INFO  app[][o.s.a.SchedulerImpl] SonarQube is up
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000dbc80000, 43515904, 0) failed; error='Cannot     allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 43515904 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/sonarqube-6.7.2/hs_err_pid10967.log
2019.05.14 09:56:08 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process     exited with exit value [ce]: 1
2019.05.14 09:56:08 INFO  app[][o.s.a.SchedulerImpl] Process [ce] is stopped
2019.05.14 09:56:11 INFO  app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.05.14 09:56:11 INFO  app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.05.14 09:56:11 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
2019.05.14 09:56:11 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
<-- Wrapper Stopped

We expect a Sonarqube server up and running without insufficient memory for the JRE to continue. If you need the dump of JVM in /opt/sonarqube-6.7.2/hs_err_pid10967.log, I can also provide it if this can help to debug the memory issue.

Thanks for your help and support.

Answer 1

It looks like the memory allocated for Compute Engine is not sufficient. In order to fix it, you'll need to upgrade to at least a t2.large and increase the size allocated for Compute Engine to 2 or 4Gb of memory.