Tomcat not using all cores in high load period

Question

We have a strange behaviour with our tomcat server. Initial situation:

• tomcat-server with 50 webapps (spring applications delivering websites using thymeleaf template engine)
• server with 16 cores and 96GB ram
• jmeter for load-testing against one webapp, others idling
  • 120 threads
  • 10 repeats
  • http requests against a bunch of urls

When we do a load test against one webapp (the others are idling) we can see that the tomcat doesn´t use all available resources. After a few seconds the cpu usage drops down to around 7% (1 core of 16). Then it rises but after some time it goes down to 7% again and stays there until all requests have been processed. If we try to open these urls in browser during the 7% period we also have very slow response times. We have no clue why tomcat is not using the other cores. On system level we can see that no other application is using the cpu at that time.

When we reduce the amount of webapps to 35 or less we see a different picture. The cpu usage now is 70% to 100% until all requests are processed and then a drop to around 0,5%.

At the moment we have no idea on where to find the reason for this behaviour. In the heap dump we can see that some thymeleaf expressions are being processed but as the “normal” requests are still being processed we don´t think that this is the problem.

Configuration:

<Connector port="8080" protocol="HTTP/1.1" maxThreads="10000" minSpareThreads="200" connectionTimeout="20000" acceptorThreadCount="4" redirectPort="8443" URIEncoding="UTF-8" />

-Djava.rmi.server.hostname=XXX
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=XXX
-Dcom.sun.management.jmxremote.access.file=XXX
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=9090
-Dcom.sun.management.jmxremote.rmi.port=9090
-Djava.awt.headless=true
-XX:PermSize=1024M
-XX:MaxPermSize=4096M
-XX:ReservedCodeCacheSize=512m
-Xdebug
-Xrunjdwp:server=y,transport=dt_socket,address=8081,suspend=n
-verbose:gc
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-Xloggc:/var/log/tomcat7/tomcat-gc.log
-Xms40000M
-Xmx40000M
-Dfile.encoding=UTF-8
-Dcatalina.base=/usr/share/tomcat7
-Dcatalina.home=/usr/share/tomcat7
-Djava.endorsed.dirs=
-Djava.io.tmpdir=/var/cache/tomcat7/temp
-Djava.util.logging.config.file=/usr/share/tomcat7/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

Apache Tomcat/7.0.39 | Java 1.7.0_75-b13

Thank you for your help!!

Answer 1

Sounds curious, but we set a tomcat start parameter to prefer IPv4, to solve this behaviour:

-Djava.net.preferIPv4Stack=true

Reference: http://www-01.ibm.com/support/docview.wss?uid=nas8N1011363

Additionally it should be helpful to configure another garbage collection:

-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC