CODEWITHSUNDEEP
performancetomcatcpu-usagetomcat8profiler
gvasquez
gvasquez

Reputation: 2007

Tomcat spiking and consuming all available CPU

Tomcat (8.0.14-1+deb8u1) in production server suddenly hogs all CPU, what could be done to diagnose the cause? I'm aware that a profiler might come in handy, but not in a production environment, any ideas?

Additional information: "top" command execution / lsb_release

Upvotes: 1

Views: 1112

Answers (2)

antoine
antoine

Reputation: 1

we are having the same issue on a couple of our servers. the problem also arises on tomcat servers without web apps, just serving static pages. cpu usage suddenly spins to 200% and never comes down

this issue never appeared before our recent upgrade to 8.0.14 and 8_u_121

setup as follows:

  • debian 8.1, latest patches
  • tomcat 8.0.14
  • oracle jvm 8_u_121 (latest as of this date)
  • ram of 2,4 and 8 GBs (depending on servers)
  • http/1.1 connector on 80 redirected to 443; nio1 connector on 443 with https (actually on 8080 and 8443, firewall maps 80 to 8080 and 443 to 8443)

when the problem occurs the symptoms are as follows:

  • cpu usage 200% by process "java", doesn't come down
  • ram consumption normal
  • no log entry in apps.log
  • about 20 entries in catalina.out as follows:

30-Jan-2017 10:15:38.479 INFO [http-nio-8080-exec-16] org.apache.coyote.http11.AbstractHttp11Processor.process Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. 30-Jan-2017 10:15:40.022 INFO [http-nio-8080-exec-17] org.apache.coyote.http11.AbstractHttp11Processor.process Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. 30-Jan-2017 10:15:41.574 INFO [http-nio-8080-exec-18] org.apache.coyote.http11.AbstractHttp11Processor.process Error parsing HTTP request header

we had similar error messages in catalina.out when we upgraded to 8.0.14. It was because this release no longer accepted unescaped { and } in url. We fixed it by changing our client code to encode these characters into %4B and %4D. There was no cpu issue at the time

Upvotes: 0

Svetlin Zarev
Svetlin Zarev

Reputation: 15673

@Gonzalo I've seen you've provided almost all necesary information, but I'll write my answer from the "beginning", so it can be helpfull for less experienced users who stumble upon it.

  1. You should take a thread dump. You can do that by either:

    • kill -3 PID
    • jstack -F PID

  2. Take a look at top or htop command output.

    • For top press "shift + h" to show the threads (You can also start top with top -p PID to filter out the irrelevant processes)
    • Find which thread consumes the CPU and write down the thread ID
    • The thread ID is in decimal, so you have to convert it to hexadecimal.

  3. Open the thread dump and look for the hexadecimal thread it. This is the thread that consumes the CPU

PS: Your thread dump looks really strange and it misses the thread IDs. The thread stacks should start like :

"Finalizer" daemon prio=8 tid=0x02b3d000 nid=0x898 in Object.wait() [0x02d0f000]
...stack traces here...

where the nid is the thread id hexadecimal you found from top/htop.

Try taking the dump with kill -3

Upvotes: 2

Related Questions