CODEWITHSUNDEEP
javatomcatliferay
SpikerTom
SpikerTom

Reputation: 125

Out of memory with tomcat 6 on RedHat but not in debian

I am using liferay 6.0.6 with tomcat 6 and using terracotta 3.5.4. The terracotta installation in on another server and works fine.

The debian and redhat systems are virtual machines and use the exact same virtual hardware. 1 cpu , 4gb ram, both 64bit OS.

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.3) (rhel-1.48.1.11.3.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

If im using a debian system and boot the application then it everything works as expected.

If im using the redhat system then i get an error :

java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:657)
    at java.util.Timer.<init>(Timer.java:176)
    at com.tc.object.locks.ClientLockManagerImpl.<init>(ClientLockManagerImpl.java:39)
    at com.tc.object.StandardDSOClientBuilder.createLockManager(StandardDSOClientBuilder.java:190)
    at com.tc.object.DistributedObjectClient.start(DistributedObjectClient.java:639)
    at com.tc.object.bytecode.ManagerImpl$2.execute(ManagerImpl.java:263)
    at com.tc.lang.StartupHelper.startUp(StartupHelper.java:39)
    at com.tc.object.bytecode.ManagerImpl.startClient(ManagerImpl.java:281)
    at com.tc.object.bytecode.ManagerImpl.init(ManagerImpl.java:202)
    at com.tc.object.bytecode.ManagerImpl.init(ManagerImpl.java:190)
    at com.tc.object.bytecode.hook.impl.DSOContextImpl.createStandaloneContext(DSOContextImpl.java:179)
    at org.terracotta.express.StandaloneL1Boot.call(StandaloneL1Boot.java:190)
    at org.terracotta.express.ClientImpl.<init>(ClientImpl.java:309)
    at org.terracotta.express.ClientFactoryImpl.newClient(ClientFactoryImpl.java:232)
    at org.terracotta.express.ClientFactoryImpl.createClient(ClientFactoryImpl.java:225)
    at org.terracotta.express.ClientFactoryImpl.createClient(ClientFactoryImpl.java:212)
    at org.terracotta.express.ClientFactoryImpl.getOrCreateClient(ClientFactoryImpl.java:190)
    at org.terracotta.express.ClientFactory.getOrCreateClient(ClientFactory.java:28)
    at net.sf.ehcache.terracotta.StandaloneTerracottaClusteredInstanceFactory.<init>(StandaloneTerracottaClusteredInstanceFactory.java:37)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

The weirdest part of this error is that memory is not used. I have set the Xmx at 3500m but the memory usage never climbs above 2.3gb.

I've been searching for a possible cause and found this : Low Java single process thread limit in Red Hat Linux

So i changed my ulimit -n and ulimit -u to 80000 but still nothing changes.

I then edited my /etc/security/limits.conf

*       soft    nproc           81920
*       hard    nproc           81920
*       soft    nofile          81920
*       hard    nofile          81920

I also edited /etc/sysctl.conf

fs.file-max = 100000

then i rebooted.

I also added -Xms=2g to my setenv.sh

I increased -Xss to -Xss5000k

I run echo 200000 > /proc/sys/kernel/threads-max

But the same error still comes up. Any ideas ?

free -m at begining of booting

             total       used       free     shared    buffers     cached
Mem:          3963       1027       2935          0         25        748
-/+ buffers/cache:        253       3709
Swap:         2047         18       2029

free -m just before crash

             total       used       free     shared    buffers     cached
Mem:          3963       2897       1065          0         31        823
-/+ buffers/cache:       2043       1920
Swap:         2047         18       2029

Update : i also tried with a machine with 8gb ram and the same result appears.

Upvotes: 1

Views: 1976

Answers (1)

Taky
Taky

Reputation: 5344

Java process contains several memory pool. java memory structure

In your case OutOfMemory error say: no enough space in your Java Process for stack allocation.

  • So if you increase -Xss pool you are able to start less threads count in your application. Try to decrease -Xss parameters. Thread with big stack you are able to start with special constructor.
  • Also you are able to decrease memory for heap, because the more memory is allocated for the heap (not necessarily used by the heap), the less memory available in the stack.
  • Or even better try to review your application structure and decrease thread count in your application, using ThreadPool or e.t.c.

Upvotes: 2

Related Questions