CODEWITHSUNDEEP
javajvmjstackjstat
Matthew Zhang
Matthew Zhang

Reputation: 1

analysis of a java process "not responding"?

My java process stopped responding. I tried to jstack but failed with below error.

21039: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Then I used -F option, but "No deadlocks found."

Other info:

java version: java version

jmap: jmap

jstat: jstat

jinfo: jinfo

Can anyone help have a look and share some links on troubleshooting this kind of java "not responding" issue?

Upvotes: 0

Views: 2855

Answers (1)

apangin
apangin

Reputation: 98630

Possible reasons of Unable to open socket file problem:

  1. The target PID is not a HotSpot JVM process.
    This is obviously not your case, since jinfo PID works fine.
  2. JVM is started with -XX:+DisableAttachMechanism option.
    This can be also verified by jinfo PID.

  3. Attach socket /tmp/.java_pidNNN has been deleted.
    There is a common practice to clean /tmp automatically with some scheduled script. In this case you should configure the cleanup software not to delete .java_pid* files.

    How to check: run lsof -p PID | grep java_pid
    If it lists a socket file, but the file does not exist, then this is exactly the described problem.

  4. Credentials of the current user (euid/egid) do not match the owner of the attach socket. Make sure you run jstack by the same user as JVM. Attach won't work if you run jstack by a different user, even if this user is root.

  5. /tmp directory of the target process is not the same as /tmp of your shell. This may happen in the following cases:
    • JVM is started in a different mount namespace. Typically this happens when JVM runs in a Docker container. Running jstack from within the same container will help.
    • JVM is started in chroot environment. For example, LXC containers may use chroot.
      How to check: run readlink -f /proc/PID/root/tmp to see if it points to /tmp or to some other directory.

  6. Current working directory of the target JVM belongs to a file system that does not allow to change permissions. CIFS and DrvFs (WSL) are examples of such file systems.

    How to check: run umask 077; touch /proc/PID/cwd/somefile.tmp, then verify that file owner is yourself, and file permissions are 600.

  7. JVM is busy and cannot reach a safepoint. For instance, JVM is in the middle of long-running garbage collection.

    How to check: run kill -3 PID. JVM should print a thread dump and heap info in its console. If JVM does not dump anything, but the process consumes almost 100% CPU or shows high I/O utilization, then this looks like the described problem.

  8. JVM process is suspended.

    How to check: run ps PID. The STAT column of alive JVM process should be Sl.

More about the internals of jstack.

There is also jattach project which is a better alternative to jstack / jmap. It can automatically handle credentials issue, it works with Docker containers, supports chroot'ed JVMs and handles uncommon file systems.

Upvotes: 4

Related Questions