CODEWITHSUNDEEP
javatomcatgarbage-collectionjvmjava-6
Ghgdasdr Bhghkjhjk
Ghgdasdr Bhghkjhjk

Reputation: 21

jvm hang and kill -3 & jmap failed

I have a tomcat process that is:

  1. all the threads is not running, all incoming connections is stuck on SYNC_RECV status
  2. not responding to kill -3

  3. jmap and jstack failed to attach to it 

    sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jmap -heap 19938
Attaching to process ID 19938, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process

sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jmap -heap -F 19938

Attaching to process ID 19938, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process

  4. the output of gc stat is not changing, even the timestamp

    sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jstat -gc -t 19938 1000 5

Timestamp        S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
       370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
       370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
       370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
       370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
       370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301

Environment information;

  1. linux

    Linux xxxx 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

  2. jdk

    java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

  3. JAVA_OPTS

    -server -Xms9g -Xmx9g -Xss256k -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=24 -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/tomcatdump  
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/tmp/tomcatlog.log -XX:NewSize=3g -XX:MaxNewSize=3g -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true

UPDATES

The SYN_RECV status of connections is a problem of LVS, I think LVS detected the abnormal of tomcat and switch the service to other server, causing the last ACK delivered to another server, then the connections on this server will stuck on SYN_RECV status, but I don't see how that relevant to the jvm hang, huge thanks to @Stephen C

UPDATES2

This process has been stuck on this status for over a week, and the CPU usage is very low

Upvotes: 1

Views: 1759

Answers (1)

Stephen C
Stephen C

Reputation: 718768

For 1) ... that could be a networking problem; e.g. https://serverfault.com/questions/273807/all-connections-from-this-network-get-stuck-in-syn-recv-state-connections-from.

For 2) not sure. However note that the standard launch script for Tomcat on Linux redirects the standard output and standard error to a log file. Look for the thread dumps in the "catalina.out" log file.

For 3) ... according to https://stackoverflow.com/a/2943651/139985 you need to run jmap / jstack as the same user that launched the JVM. It is probably not root.

For 4) ... that could be explained by the fact that the GC hasn't needed to run.

UPDATES

It is also possible that SELinux may be getting in the way.

I'm afraid that while it might appear to you that the network is properly configured, there is definitely something wrong at that level. The SYN_RECV state is a state in TCP "3-way handshake" that happens when a connection is being established in the network stack. Java is not involved in that processes. Basically you have lots of Java threads that have attempted to initiate Socket connections, but the connects have all jammed up.

Upvotes: 1

Related Questions