Severe Tomcat crash has me stumped

Question

I'm pretty used to crashing my tomcat server. But I don't think I made any changes at all and seem to have broken it completely. Was working one min then not the next.

I really hope someone can help here is the Catalina.out:

Dec 20, 2012 1:35:56 AM org.apache.catalina.startup.Embedded initDirs
SEVERE: Cannot find specified temporary folder at 
Dec 20, 2012 1:35:57 AM org.apache.catalina.core.AprLifecycleListener init
INFO: Loaded APR based Apache Tomcat Native library 1.1.23.
Dec 20, 2012 1:35:57 AM org.apache.catalina.core.AprLifecycleListener init
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
Dec 20, 2012 1:35:57 AM org.apache.coyote.http11.Http11AprProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-8040
Dec 20, 2012 1:35:57 AM org.apache.coyote.ajp.AjpAprProtocol init
INFO: Initializing Coyote AJP/1.3 on ajp-8042
Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 587 ms
Dec 20, 2012 1:35:57 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Dec 20, 2012 1:35:57 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.33
Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory docs
Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ROOT
Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory myApp
Dec 20, 2012 1:35:57 AM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/lhome/me/tomcat6/webapps/myApp/WEB-INF/lib/javax.servlet.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
Dec 20, 2012 1:35:58 AM org.apache.coyote.http11.Http11AprProtocol start
INFO: Starting Coyote HTTP/1.1 on http-8040
Dec 20, 2012 1:35:58 AM org.apache.coyote.ajp.AjpAprProtocol start
INFO: Starting Coyote AJP/1.3 on ajp-8042
Dec 20, 2012 1:35:58 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 880 ms
Error Reading TLE line 1: java.lang.Exception: TLE line 1 not valid first line
Error Creating SGP4 Satellite
Dec 20, 2012 1:36:26 AM org.apache.coyote.http11.Http11AprProtocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8040
Dec 20, 2012 1:36:26 AM org.apache.coyote.ajp.AjpAprProtocol pause
INFO: Pausing Coyote AJP/1.3 on ajp-8042
Dec 20, 2012 1:36:27 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Dec 20, 2012 1:36:30 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/myApp] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Dec 20, 2012 1:36:30 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/myApp] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
Dec 20, 2012 1:36:30 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/myApp] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
Dec 20, 2012 1:36:30 AM org.apache.coyote.http11.Http11AprProtocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8040
Dec 20, 2012 1:36:30 AM org.apache.coyote.ajp.AjpAprProtocol destroy
INFO: Stopping Coyote AJP/1.3 on ajp-8042
Dec 20, 2012 1:37:12 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop: 
java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
    at java.net.Socket.connect(Socket.java:579)
    at java.net.Socket.connect(Socket.java:528)
    at java.net.Socket.<init>(Socket.java:425)
    at java.net.Socket.<init>(Socket.java:208)
    at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:422)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:338)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:416)

And I have a new log that has been created, I've never seen it before but it seems to think its something severe. A long one but here it is:

Massive Error Log I've never seen before

Answer 1

Thanks to all who gave help here.

However I fell I should answer my own question. Although I still don't fully understand why it caused such a severe exception. The problem seemed to lie with me having dropped a table on my database that the program was reading. This table was full of data that the program used to compile other data. Once I re-populated this table it was fine.

I don't know why something so simple caused such a lot of errors and never showed up in the logs as a table not found error. However I can only assume that the program was trying to run with blank data and this someone caused a severe error.

So thanks again to those who offered help!

Answer 2

The bomb seems to have occurred in the native method java.util.zip.ZipFile.getZipMessage() which is called by getEntry(). ZipFile had experienced an error and was trying to construct an exception message. This occurs while stopping the running webapp as part of reload.

Did you move the .war file? That can cause the problem.

These kinds of problems are hard to diagnose and fix. However, the problem is very likely to be java.io.tmpdir not being set properly. http://efod.se/blog/archive/2011/05/02/java-sigbus is a similar issue.

If that doesn't work, use apt-get to upgrade zlib and OpenJDK, and force ZipFile.getEntry() to be interpreted:

-X:CompileCommand=exclude,java/util/zip/ZipFile,getEntry

As OSB Stats said, don't trust your hardware (although it's not likely a hardware problem).

Answer 3

Your JVM's got SIGBUS. This is either a hardware problem, or a low-level JVM code bug.

siginfo:si_signo=SIGBUS: si_errno=0, 
si_code=2 (BUS_ADRERR), si_addr=0x00007f0a9bb06fe5

One suspicious entry is -Djava.io.tmpdir= in your startup (i.e. temp dir is empty string). Could do funny things to JVM, though I did not test it. May be it is OK to have it like that.

jvm_args: -Dcatalina.base=/lhome/me/tomcat6 
-Dcatalina.home=/usr/share/tomcat6 
-Djava.endorsed.dirs= 
-Djava.io.tmpdir= 
-Djava.util.logging.config.file=/lhome/me/tomcat6/conf/logging.properties 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 

Bottom line: I believe your hardware goes down. Backup everything.