CODEWITHSUNDEEP
javalinuxjvmtomcat6
Oreli Levi
Oreli Levi

Reputation: 71

Tomcat process suddenly crushes

Trying to diagnose some bizarre Tomcat 6 and/or JVM errors on a 64-bit linux machine:

OS VERSION : RED HAT ENTERPRISE LINUX SERVER RELEASE 6.3 (SANTIAGO)
SERVER MODEL : VMWARE VIRTUAL PLATFORM ()

I am running some application tests on virtual machines and facing a weird issue. The problem occures on certain VMs where exactly 2 process are running: Tomcat Application server process (java process) and a monitor process, at one randomal point, after some random hours of execution the following message can be found on catalina.out (server log) of the VM that fails (the problem doesnt occurs always on same machine):

org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-50100
INFO: Stopping service Catalina
org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080

After this moment the Tomcat process is shutted down and execution fails. No picks can be seen on memory or CPU use. No system messages can be seen under /var/logs.

  1. At first stage we suspected that an external signal is killing our Tomcat process and we tried to use the -Xrs flag for JVM arguments of the process but issue still occurred.

    The java flag (-xrs ) should prevent the Java runtime environment handling exception signals such SIGSEGV and SIGABRT not help

    Before adding xrs flag we saw INFO: Pausing Coyote HTTP/1.1 on http-50100 error that happened because outcome signal

    After adding `xrs flag the kettle/tomcat behaves the same way and crushes without any errors in log files.

  2. Running starce command on relevant process gives us the following output:

    Process 12917 attached - interrupt to quit futex(0x7f6b9e8ab9d0, FUTEX_WAIT, 12918, NULLPANIC: attached pid 12917 exited with 143

    We cannot run starce -f to include all threads because it causes our application to be stuck. However, killing son process with kill command behaves similary - we see that parent JVM exit with error code 143 and we see also the signal sent to the son process.

  3. We are running also with java -XX:+HeapDumpOnOutOfMemoryError optind to create a heap dump for jvm crash due to out of memory error but no core dump or heap dumps are creted due to the crush.

Any ideas?

Upvotes: 4

Views: 2607

Answers (1)

Alberto Pires
Alberto Pires

Reputation: 319

is it possible, that before the crash, there is some sort of memory depletion, so that the java process could be killed by the OOM. Also, I found a link http://bugs.centos.org/view.php?id=4059 that describes a similar problem.

Upvotes: 0

Related Questions