CODEWITHSUNDEEP
javatomcatgarbage-collectionjvm
Kev0815
Kev0815

Reputation: 1

Freezes of JVM/Tomcat under load while safepoint-log logs thausends safepoints "ThreadStop" entries

I have a problem with a Java application running in a Tomcat running with openJDK11. The problem is that after some time under high load, the application freezes. The CPU load then also collapses and no more entries are written in Catalina.out or application log files. After some time (1-2 minutes) the system recovers and runs normally.

I first thought that there was a problem with the garbage collection and started the logging for garbage collections and safepoints.

In the Safepoint log there are hundreds to thousands of entries (without interruption of other messages) at freeze times:

[2021-02-03T08:53:44.619+0100][5136.564s][info][safepoint] Application time: 0.0153146 seconds
[2021-02-03T08:53:44.620+0100][5136.565s][info][safepoint] Entering safepoint region: ThreadStop
[2021-02-03T08:53:44.621+0100][5136.566s][info][safepoint] Leaving safepoint region
[2021-02-03T08:53:44.621+0100][5136.566s][info][safepoint] Total time for which application threads were stopped: 0.0020579 seconds, Stopping threads took: 0.0006463 second

The ThreadStop messages correspond 1:1 with the times of the freezes. Therefore the assumption that the ThreadStops are the cause for the problem.

Unfortunately I could not find anything about ThreadStop Safepoints.

Therefore the question, what triggers ThreadStop Safepoints? Are they the cause of the freezes? If so, how can I solve the problem?

Thanks a lot

Upvotes: 0

Views: 471

Answers (1)

apangin
apangin

Reputation: 98380

ThreadStop is a VM operation, performed at a global safepoint, to pass an asynchronous exception to some thread.

As of JDK 11, there are two common reasons for such operation:

  • Thread.stop method that delivers an asynchronous ThreadDeath error to the given thread;
  • JVM TI StopThread function.

async-profiler will help to find where ThreadStop is initiated from.

To start profiling asynchronous exceptions, run

./profiler.sh start -e Thread::send_async_exception <PID>

Then, after one or more ThreadStop happened, stop profiling and get the results:

./profiler.sh stop -f result.html <PID>

ThreadStop profile

For example, here we see that Tomcat calls Thread.stop when reloading a web application. This happens, if an application does not gracefully stop all its threads on its own, and clearReferencesStopThreads attribute is set to true. See Context Configuration Reference for details.

Upvotes: 1

Related Questions