CODEWITHSUNDEEP
unixmemory-leakslog4jtomcat7servlet-3.0
ddoor
ddoor

Reputation: 5963

tomcat java.util.logging process

I have a linux (ubuntu) server running tomcat7. The server has 1GB of ram, and I am getting heap space errors thrown from java.

Basically what is happening, is that my tomcat server (running my servlet code) is throwing exceptions because there is no more memory to allocate. My code is pretty lean, so I don't think that is the problem.

When I look at htop, I see a lot of java logging processes, I see about 30 of these processes:

ps ax | grep java
 7412 pts/0    Sl     0:02 /usr/bin/java -Djava.util.logging.config.file=/usr/share/tomcat/apache-tomcat-7.0.29/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat/apache-tomcat-7.0.29/endorsed -classpath /usr/share/tomcat/apache-tomcat-7.0.29/bin/bootstrap.jar:/usr/share/tomcat/apache-tomcat-7.0.29/bin/tomcat-juli.jar -Dcatalina.base=/usr/share/tomcat/apache-tomcat-7.0.29 -Dcatalina.home=/usr/share/tomcat/apache-tomcat-7.0.29 -Djava.io.tmpdir=/usr/share/tomcat/apache-tomcat-7.0.29/temp org.apache.catalina.startup.Bootstrap start
 7426 pts/0    D+     0:00 grep --color=auto java
25937 ?        Sl    13:12 /usr/bin/java -Djava.util.logging.config.file=/usr/share/tomcat/apache-tomcat-7.0.29/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat/apache-tomcat-7.0.29/endorsed -classpath /usr/share/tomcat/apache-tomcat-7.0.29/bin/bootstrap.jar:/usr/share/tomcat/apache-tomcat-7.0.29/bin/tomcat-juli.jar -Dcatalina.base=/usr/share/tomcat/apache-tomcat-7.0.29 -Dcatalina.home=/usr/share/tomcat/apache-tomcat-7.0.29 -Djava.io.tmpdir=/usr/share/tomcat/apache-tomcat-7.0.29/temp org.apache.catalina.startup.Bootstrap start

These java processes slowly leaks memory over time, and because I only have 1gb of ram to play with my tomcat server starts throwing heap space exceptions.

I have fiddled with log4j (but it should be turned off) and I have done all I can to turn off logging in configs I can find.

Any help would be great, I just need to get rid of these processes - they are way to greedy. Thanks!

~ Dan

EDIT: MORE INFO:

This is probably a big enough hint to solve this one entirely - I checked some of the startup debug and came up with this:

    Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-53] but has failed to stop it. This is very likely to create a memory leak.
Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads

SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-54] but has failed to stop it. This is very likely to create a memory leak.
Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads

SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-62] but has failed to stop it. This is very likely to create a memory leak.
Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads

SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-64] but has failed to stop it. This is very likely to create a memory leak.

One problem with leaks that I found is due to a bug with the JDBC Driver http://bugs.mysql.com/bug.php?id=36565 according to a couple of forums. That solves another message I was getting.

I have been looking around and I have found this which is quite useful:

http://wiki.apache.org/tomcat/MemoryLeakProtection

Now I am just trying to work through all your advise thanks heaps!

~ Dan

Upvotes: 3

Views: 4546

Answers (2)

Anthony
Anthony

Reputation: 12736

Tomcat usually has a thread pool. The Java EE specification requires to start a separate thread for processing each HTTP request. Tomcat has a thread pool and reuses threads from this pool instead of creating and terminating them. More details are available here

You can set the max memory limit for Tomcat by modifying setenv.sh (or setenv.bat respectively) eg:

CATALINA_OPTS='-Xms512m -Xmx1024m'

More details for the Oracle JVM are available here

But in practice the most important thing is to track memory allocation inside the JVM. You will need Java VisualVM for this.

Upvotes: 1

pd40
pd40

Reputation: 3247

I general I would expect the logging code in java to have been well tested. If there is no log output being generated by your logging config it has a low overhead. Your htop results are probably showing some idle thread status in tomcat.

It could be logging but you may want to consider all the ways your application or another component may be leaking memory.

All JVM's have pretty good tools for diagnosing memory leaks. Threads like this discuss great ways to find out what is going on.

Edit: some more thoughts

You might want to check your configuration for too many sessions if a lot of requests are using HttpServletRequest.getSession(). The Manager component config goes inside Context and defaults to -1 for unlimited sessions.

Check your version of tomcat with the current issue fix list in case you are encountering a known memory leak.

Lastly, Tomcat 7 includes a JreMemoryLeakPreventionListener that might show resources not being released.

Upvotes: 3

Related Questions