tomcat6: memory leak when stopping

Question

Every time when I stop tomcat6 I get memory leak. This is from my catalina.out:

Mar 10, 2014 5:21:01 PM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
Mar 10, 2014 5:21:02 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
17:21:02,852  INFO XmlWebApplicationContext:696 - Closing org.springframework.web.context.support.XmlWebApplicationContext@1ba6b028: display name [WebApplicationContext for namespace 'documation-servlet']; startup date [Mon Mar 10 17:17:34 EDT 2014]; parent: org.springframework.web.context.support.XmlWebApplicationContext@57af0af7
17:21:02,853  INFO DefaultListableBeanFactory:282 - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@fb7f97b: defining beans [propertyConfigurer,methodNameResolver,exportController,managerController,urlMapping,viewResolver]; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@2e2e1b6c
17:21:02,938  INFO XmlWebApplicationContext:696 - Closing org.springframework.web.context.support.XmlWebApplicationContext@57af0af7: display name [Root WebApplicationContext]; startup date [Mon Mar 10 17:17:28 EDT 2014]; root of context hierarchy
17:21:02,938  INFO DefaultListableBeanFactory:282 - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2e2e1b6c: defining beans [propertyConfigurer,documationDataSource,upsDataSource,securityFilter,documationXactionManager,aclService,libraryService,librarySearchService,inspectionService,approvalService,resourceMgr,zkService,mailerService,libraryServiceTX,librarySearchServiceTX,zkServiceTX,approvalServiceTX,inspectionServiceTX,aclServiceTX,appCacheManager,methodCachingAdvice]; root of factory hierarchy
17:21:02,939  INFO EhCacheManagerFactoryBean:143 - Shutting down EHCache CacheManager
Mar 10, 2014 5:21:02 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [Timer-2] but has failed to stop it. This is very likely to create a memory leak.
Mar 10, 2014 5:21:02 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [Resource Destroyer in BasicResourcePool.close()] but has failed to stop it. This is very likely to create a memory leak.
Mar 10, 2014 5:21:02 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [Resource Destroyer in BasicResourcePool.close()] but has failed to stop it. This is very likely to create a memory leak.
Mar 10, 2014 5:21:02 PM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080

I am using spring framework and zk framework for my application. Any idea what can cause this?

Answer 1

I don't know about "Timer-2", but "Resource Destroyer in BasicResourcePool.close()" is a thread started by the close method of the c3p0 connection pool. It does some cleanup actions and dies naturally after its work is done, so you shouldn't worry about a real memory leak here. If "Timer-2" on the other hand is a deamon thread (judged by the name, this could well be the case), then it is worthwhile to investigate why it isn't stopped on application shutdown.

Answer 2

Contrary to what @Garreth says, you quite likely have a memory leak. If you never re-deploy tour web application and always shut-down the JVM between deployments, then it is likely that it does not make a difference at all. But it's nice to have a re-deployable web application, right?

You have one or more components that are launching threads at some point that are not cleaning them up. BasicResourcesPool is something you can probably find rather quickly by looking in your own code and the classes provided by any libraries you are using.

Timer-2 is more difficult to track-down because it has such a generic name.

If you have the opportunity to attach a profiler, you can track object and/or thread creations and see what component is starting those threads. Whatever component (listener, servlet, etc.) that started those problematic components needs to shut them down properly when the application is being undeployed. Consider using Servlet.destroy or ServletContextListener.contextDestroyed to perform such cleanup operations.

Answer 3

Tomcat is paranoid about memory and regularly warns about potential memory leaks. The warning says you "may" cause a leak, not that you definitely have one.

You need to track memory usage on the server to see if you actually have a leak. Note that the JVM that runs Tomcat will consume most of the memory on a given server, and allocate to other processes if they need them. As such, don't be unnecessarily concerned if you take a snapshot and see that your server has very little "free" memory.

A leak will occur if you application is committing something to memory that Garbage Collection does not clear, because its flagged for non-deletion.

To expose this, track the memory being used by the tomcat user.

ps -u tomcat u | awk '{sum +=$4}; END {print sum}'

will tell you what % of available memory the tomcat user has reserved.