Out of memory in maven on unpriviledged user

Question

I am experiencing strange maven problem when running our maven tests on a CentOS box. The thing is, when I run

mvn install

as unpriviledged user, it hangs forever with following exception

java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:713)
at org.apache.maven.surefire.booter.ForkedBooter.launchLastDitchDaemonShutdownThread(ForkedBooter.java:181)
at org.apache.maven.surefire.booter.ForkedBooter.exit(ForkedBooter.java:141)
at org.apache.maven.surefire.booter.ForkedBooter.exit(ForkedBooter.java:141)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:125)

Exception in thread "main" Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native threadjava.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:713)
at org.apache.maven.surefire.booter.ForkedBooter.launchLastDitchDaemonShutdownThread(ForkedBooter.java:181) at org.apache.maven.surefire.booter.ForkedBooter.launchLastDitchDaemonShutdownThread(ForkedBooter.java:181)
at org.apache.maven.surefire.booter.ForkedBooter.exit(ForkedBooter.java:141)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:133)

Exception in thread "metrics-meter-tick-thread-1" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:713)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)  at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
at java.lang.Thread.run(Thread.java:744)

However if I run the mvn as root, there's no problem. Also, on an Ubuntu box as unpriviledged user, the tests run fine.

My Stack:

• Maven occurs both on 3.0.5 and 3.1.1
• CentOS 6.5 kernel 2.6.32-431.3.1.el6.x86_64
• Ubuntu 13.10 kernel 3.11.0-15-generic
• JDK 1.7.0_51 (both on ubuntu and CentOS)

All is pretty much vanilla.

Other facts:

• $MAVEN_OPTS are not set in any shell (either CentOS or Ubuntu). I did try combinations of MAVEN_OPTS="-Xms256m -Xmx4096m -XX:MaxPermSize=256m -Dmaven.surefire.debug=-Xmx3000m" but with same result
• $JAVA_OPTS are not set in any shell neither. I tried similar options as above, and since it had no effect, I dropped it.
• tried to parametrize the surefire plugin in ~/.m2 directory for every user I tried but without any result, so I dropped the config.
• from pstree I see the maven's java creates forks to ~1060 procsses in all cases.

Can somebody please explain what's going on.

Many thanks

Answer 1

java.lang.OutOfMemoryError: unable to create new native thread

This is the important part. OutOfMemoryError has been expanded to mean 'cannot allocate some basic resource'. In this case, new threads, not memory.

Probable explanation: your 'ulimit' on the number of threads is smaller for a regular user than for root. When you went to a new Linux, it had a larger maximum for regular users.

I found this question because the "metrics-meter-tick-thread-1" threads are causing me major grief- a (Cassandra) library is creating these over and over again and I have 300 threads named "metrics-meter-tick-thread-1" and "metrics-meter-tick-thread-2".

Answer 2

Well not exactly an answer...

I finally moved to Ubuntu server and everything is running smootly. I repeat, both installations are vanilla, out-of-the-box, no customization (doesn't mean I didn't try any). I reinstalled the CentOS box twice to make sure I didn't left any misconguration. So for our tests, I wouldn't recommend CentOS for maven...

Feel free to answer anyway if you have any thoughts on what's going on. I can change the accepted answer anytime.

Answer 3

I have a setup similar to yours, except that I'm running on CentOS 6.4 and I'm using jdk 1.6. In order to get Maven to work on largish builds I had to set

export MAVEN_OPTS="-Xms256M -Xmx1024M -XX:MaxPermSize=512M"

I put this and other Maven related settings in a mvn.sh file in the /etc/profile.d directory, as I'm using bash.