CODEWITHSUNDEEP
javajvm
jonderry
jonderry

Reputation: 23633

Java process stuck in SIGTERM handler deleting a file

I have a java process that is failing to respond to a SIGTERM. This is a sporadic issue but occurred on several servers at the same time. I know that I can just kill it with SIGKILL but want to understand why it may be stuck not responding to the SIGTERM signal(s) that have been sent to it. Looking at the output of jstack, it seems the following threads are stuck:

"SIGTERM handler" daemon prio=10 tid=0x00007fcb30006000 nid=0x259c waiting for monitor entry [0x00007fc98cd2c000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:212)
        - waiting to lock <0x00000005fc7cdc60> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:52)
        at sun.misc.Signal$1.run(Signal.java:212)
        at java.lang.Thread.run(Thread.java:745)

"SIGTERM handler" daemon prio=10 tid=0x00007fcb30005000 nid=0x30cb waiting for monitor entry [0x00007fc982282000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:212)
        - waiting to lock <0x00000005fc7cdc60> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:52)
        at sun.misc.Signal$1.run(Signal.java:212)
        at java.lang.Thread.run(Thread.java:745)

"SIGTERM handler" daemon prio=10 tid=0x00007fcb30004000 nid=0x1305 waiting for monitor entry [0x00007fc982a8a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Shutdown.exit(Shutdown.java:212)
        - waiting to lock <0x00000005fc7cdc60> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:52)
        at sun.misc.Signal$1.run(Signal.java:212)
        at java.lang.Thread.run(Thread.java:745)

"SIGTERM handler" daemon prio=10 tid=0x00007fcb30003000 nid=0xa3e runnable [0x00007fc982585000]
   java.lang.Thread.State: RUNNABLE
        at java.io.UnixFileSystem.delete0(Native Method)
        at java.io.UnixFileSystem.delete(UnixFileSystem.java:265)
        at java.io.File.delete(File.java:1035)
        at java.io.DeleteOnExitHook.runHooks(DeleteOnExitHook.java:80)
        at java.io.DeleteOnExitHook$1.run(DeleteOnExitHook.java:49)
        at java.lang.Shutdown.runHooks(Shutdown.java:123)
        at java.lang.Shutdown.sequence(Shutdown.java:167)
        at java.lang.Shutdown.exit(Shutdown.java:212)
        - locked <0x00000005fc7cdc60> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:52)
        at sun.misc.Signal$1.run(Signal.java:212)
        at java.lang.Thread.run(Thread.java:745)

There are many other threads, but nothing looks out of the ordinary in them. The thread holding the lock appears to be deleting some file. Does anyone know what this file may be and why deletion of it may be stuck?

Upvotes: 1

Views: 1021

Answers (1)

Philipp Wendler
Philipp Wendler

Reputation: 11433

It is deleting a file that was marked for deletion on JVM termination with File.deleteOnExit. This is probably a temporary file your code or a library of yours created. It is not clear to me why this could be hanging. Maybe the file is on a very slow file system (a remote file system), or some I/O device is dead?

Finding out which file this is could be complicated, if you don't find a likely candidate when looking for callers to deleteOnExit in your code.

You could check for likely candidates in the list of open files of your JVM (with lsof), but the file might not be opened currently.

Probably the best bet is to attach strace to your JVM with strace -e unlink,unlinkat -p <PID> right before sending SIGTERM and check what files the JVM attempts to delete.

Upvotes: 2

Related Questions