Requests take too much time in Tomcat 8 on peak time

Question

I have configured tomcat with the following configurations:

<Connector port="8080" protocol="HTTP/1.1"
           connectionTimeout="20000"
           maxThreads="500"
           maxConnections="20000"
           acceptCount="150"
           etc... />

same numbers for AJP connector, maxThreads=500 and acceptCount="150".

It works fine most of the time, but on peak times, when I have much more requests than usual, it takes too long to respond. Sometimes above 15 seconds and in rare cases timeOut. It may look okay, as maxThreads=500 and I have several thousand requests, however, on Server Status I see:

Max threads: 500 Current thread count: 17 Current thread busy: 1 Keep alive sockets count: 1

The max number of currentThreadCount I have seen so far was 27. If there are so many connections, shouldn't tomcat create more threads (up to 500) to respond faster?

So, what am I doing wrong? What am I missing? I have 2 core CPU (max usage during peak hours ~10%) and 2GB of RAM (max usage 60%).

Short info about web app: normally, each user makes at least 2 requests per session: static JSON response and 1 database query. In peak time I have 15-20k active users, but I don't know how many requests per second do I get. However, slow responses start from 5k active users.

I also increased max-active connections on app properties, with no change on performance, my current application.properties:

spring.jpa.hibernate.ddl-auto=update
spring.datasource.driverClassName=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/database_name
spring.datasource.username=$username$
spring.datasource.password=$password$
spring.datasource.tomcat.max-active=200
spring.datasource.tomcat.max-wait=10000
spring.datasource.tomcat.max-idle=50
spring.datasource.tomcat.min-idle=10
spring.datasource.tomcat.initial-size=10

UPDATE I changed default JDBC connection pool to Hikari with the following configurations and enabled jta, however, didn't feel any difference on peak times:

spring.jta.enabled=true
spring.datasource.hikari.maximum-pool-size=125
spring.datasource.hikari.minimum-idle=5

I am adding database query below. Results of the query later added into another object and returned as ResponseBody.

@Query("select new ObjectClass(s.id, s.a, s.b, s.c") from TableName s " +
        "where s.x > :param order by id desc")
List<ObjectClass> getObjects(@Param("param") long param);

CPU usage doesn't grow, RAM is almost half-free, if I am having too many requests, shouldn't I have overloaded on the server? Instead, I just get slow response time. Therefore, I think I have a configuration problem which I want to resolve.

-Xms512M -Xmx1024M

The app that hangs on peak time:

Active sessions: 3243 Session count: 475330 Max active sessions: 4685 Rejected session creations: 0 Expired sessions: 472105 Longest session alive time: 7457 s Average session alive time: 9 s Processing time: 3177 ms JSPs loaded: 0 JSPs reloaded: 0

Stack trace:

"Attach Listener" #502 daemon prio=9 os_prio=0 tid=0x00007fde58007800 nid=0x3ff waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Abandoned connection cleanup thread" #69 daemon prio=5 os_prio=0 tid=0x00007fde6c03e800 nid=0xa44 in Object.wait() [0x00007fde471ba000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    - locked <0x00000000c259e618> (a java.lang.ref.ReferenceQueue$Lock)
    at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"ajp-nio-8009-exec-25" #68 daemon prio=5 os_prio=0 tid=0x00007fde40016000 nid=0x741 waiting on condition [0x00007fde35fe0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c1cc6758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

"ajp-nio-8009-exec-11" #54 daemon prio=5 os_prio=0 tid=0x00007fde38041800 nid=0x733 waiting on condition [0x00007fde36fee000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c1cc6758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

"ajp-nio-8009-AsyncTimeout" #52 daemon prio=5 os_prio=0 tid=0x00007fde884e8800 nid=0x732 waiting on condition [0x00007fde370ef000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.coyote.AbstractProtocol$AsyncTimeout.run(AbstractProtocol.java:1211)
    at java.lang.Thread.run(Thread.java:748)

"ajp-nio-8009-Acceptor-0" #51 daemon prio=5 os_prio=0 tid=0x00007fde884e6800 nid=0x731 runnable [0x00007fde371f0000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
    - locked <0x00000000c019d7e8> (a java.lang.Object)
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:455)
    at java.lang.Thread.run(Thread.java:748)

"ajp-nio-8009-ClientPoller-1" #50 daemon prio=5 os_prio=0 tid=0x00007fde884e4800 nid=0x730 runnable [0x00007fde372f1000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000000c1da2fa0> (a sun.nio.ch.Util$3)
    - locked <0x00000000c1da2f90> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000c1d5b1e0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:787)
    at java.lang.Thread.run(Thread.java:748)

"ajp-nio-8009-ClientPoller-0" #49 daemon prio=5 os_prio=0 tid=0x00007fde884d6000 nid=0x72f runnable [0x00007fde373f2000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000000c1d510d8> (a sun.nio.ch.Util$3)
    - locked <0x00000000c1d510c8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000c1ce78c0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:787)
    at java.lang.Thread.run(Thread.java:748)

"ajp-nio-8009-exec-10" #48 daemon prio=5 os_prio=0 tid=0x00007fde884c7000 nid=0x72e waiting on condition [0x00007fde374f3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c1cc6758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

"ajp-nio-8009-exec-2" #40 daemon prio=5 os_prio=0 tid=0x00007fde884b7000 nid=0x726 waiting on condition [0x00007fde37cfb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c1cc6758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

"ajp-nio-8009-exec-1" #39 daemon prio=5 os_prio=0 tid=0x00007fde884b5000 nid=0x725 waiting on condition [0x00007fde37dfc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c1cc6758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

"http-nio-127.0.0.1-8080-AsyncTimeout" #38 daemon prio=5 os_prio=0 tid=0x00007fde884b3000 nid=0x724 waiting on condition [0x00007fde37efd000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.coyote.AbstractProtocol$AsyncTimeout.run(AbstractProtocol.java:1211)
    at java.lang.Thread.run(Thread.java:748)

"http-nio-127.0.0.1-8080-Acceptor-0" #37 daemon prio=5 os_prio=0 tid=0x00007fde884b1800 nid=0x723 runnable [0x00007fde37ffe000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
    - locked <0x00000000c01a03b8> (a java.lang.Object)
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:455)
    at java.lang.Thread.run(Thread.java:748)

"http-nio-127.0.0.1-8080-exec-1" #25 daemon prio=5 os_prio=0 tid=0x00007fde88324000 nid=0x717 waiting on condition [0x00007fde46db8000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c1d9c4e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" #24 daemon prio=5 os_prio=0 tid=0x00007fde88323000 nid=0x716 waiting on condition [0x00007fde476bb000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1355)
    at java.lang.Thread.run(Thread.java:748)

"Abandoned connection cleanup thread" #22 daemon prio=5 os_prio=0 tid=0x00007fde4ca72800 nid=0x6f5 in Object.wait() [0x00007fde45c22000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    - locked <0x00000000c102c4b0> (a java.lang.ref.ReferenceQueue$Lock)
    at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"Tomcat JDBC Pool Cleaner[1595428806:1507838479700]" #21 daemon prio=5 os_prio=0 tid=0x00007fde4ca5b800 nid=0x6f4 in Object.wait() [0x00007fde470b9000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.util.TimerThread.mainLoop(Timer.java:552)
    - locked <0x00000000c0f6fe80> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:505)

"NioBlockingSelector.BlockPoller-2" #13 daemon prio=5 os_prio=0 tid=0x00007fde8847e000 nid=0x66f runnable [0x00007fde478bd000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000000c019bd40> (a sun.nio.ch.Util$3)
    - locked <0x00000000c019bd30> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000c019bbf8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:339)

"NioBlockingSelector.BlockPoller-1" #12 daemon prio=5 os_prio=0 tid=0x00007fde8846f800 nid=0x66e runnable [0x00007fde479be000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000000c019ec10> (a sun.nio.ch.Util$3)
    - locked <0x00000000c019ec00> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000c019ead8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:339)

"GC Daemon" #11 daemon prio=2 os_prio=0 tid=0x00007fde883f9000 nid=0x66b in Object.wait() [0x00007fde741c6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000c02f16d8> (a sun.misc.GC$LatencyLock)
    at sun.misc.GC$Daemon.run(GC.java:117)
    - locked <0x00000000c02f16d8> (a sun.misc.GC$LatencyLock)

"AsyncFileHandlerWriter-1510467688" #10 daemon prio=5 os_prio=0 tid=0x00007fde88168800 nid=0x63e waiting on condition [0x00007fde7475c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c02f16e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
    at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
    at org.apache.juli.AsyncFileHandler$LoggerThread.run(AsyncFileHandler.java:160)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007fde880af000 nid=0x62e runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fde880ac000 nid=0x62d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fde880a9000 nid=0x62c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fde880a7000 nid=0x62b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fde88080000 nid=0x625 in Object.wait() [0x00007fde74f33000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    - locked <0x00000000c02f7408> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fde8807b800 nid=0x622 in Object.wait() [0x00007fde75034000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
    - locked <0x00000000c02f7490> (a java.lang.ref.Reference$Lock)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007fde8800a800 nid=0x589 runnable [0x00007fde8f6af000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
    at java.net.ServerSocket.implAccept(ServerSocket.java:545)
    at java.net.ServerSocket.accept(ServerSocket.java:513)
    at org.apache.catalina.core.StandardServer.await(StandardServer.java:466)
    at org.apache.catalina.startup.Catalina.await(Catalina.java:744)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:690)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:355)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:495)

"VM Thread" os_prio=0 tid=0x00007fde88073800 nid=0x5fd runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fde8801f800 nid=0x597 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fde88021000 nid=0x598 runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007fde880bd800 nid=0x62f waiting on condition 

JNI global references: 317

Update: While I haven't resolved my problem, @Per Huss's answer pushed me towards the right direction on analysing each thread separately and find the problem. I have to award my bounty now, therefore, I will award it to him. However, I thank everyone who commented here, as all comments helped me to learn something new.

Update 2: It looks like the problem is within apache. On peak times even static pages have slow response time, even the ones from other apps. Including tomcat manager. So, I changed prefork to mpm_worker and currently testing different configurations. I will update this thread with the results, soon.

Answer 1

In these kind of bugs , First of all we should identify where is problem . Here giving a plan of action for debugging these type of issues :

For example in your case

Requests come from user to tomcat , then it will give it to your application .

First of all , check where is issue i.e. there can be issues in following places:

1. Your any application api or all api started taking time, but tomcat threads are free
2. Your tomcat threads are not free and processing of these each thread is taking time, so latency occurs
3. You database starts taking time
4. As you are querying the database, there may be case that more data is being loaded into your app and some java gc issues started occurs

So, in first case , please check your application logs and if logs are not there please put the logs and check , if any of your application is taking time (Logs Never Lies )

In the second case, check your tomcat logs that what is condition there .

In third case, please check your database logs , that queries is taking more time or not .

In the fourth case, you can monitor your java health monitoring , there are many tools in market like jfr, jcisualvm etc ..

Also, your question has not enough explanation, please answer the following

1. What is sample structure of your application?
2. What you do to bring back your application not normal state, for example, restarting solves your issue or not? I am asking this because if you need to restart it, then there may be deadlock so you might need to take jstack and analyse it
3. How much XMX is given to the application?
4. Are your database server and application server on the same machine? Because there may be some io problem in peak time on some machine, so we need to check both

Please identify first where is problem, then we can proceed further, how we can identify and solve the issue .

Thanks

Answer 2

I finally solved my problem. In fact, it was apache that didn't allow enough connections. First of all, I changed prefork to mpm worker. Later, I increased the number of MaxRequestWorkers.

<IfModule mpm_worker_module>
    StartServers             2
    MinSpareThreads          50
    MaxSpareThreads          125
    ThreadLimit              64
    ThreadsPerChild          25
    ServerLimit              5000
    MaxRequestWorkers        5000
    MaxConnectionsPerChild   4500

Earlier, I was getting slow response time already with 3000 active users. With the new configuration even 17000 active users didn't increase response time and it was working like in normal times. As expected, CPU usage and RAM increased on peak time and then went back to normal.

Answer 3

If there are so many connections, shouldn't tomcat create more threads (up to 500) to respond faster?

=> As per Tomcat8 docs, if more simultaneous requests are received than can be handled by the currently available request processing threads, additional threads will be created up to the configured maximum (the value of the maxThreads attribute). If still more simultaneous requests are received, they are stacked up inside the server socket created by the Connector, up to the configured maximum (the value of the acceptCount attribute).

So your tomcat must be creating the threads as required. Also, Tomcat 8 by default works in NIO mode meaning one thread can serve multiple requests. You can confirm that behavior by starting monitoring tool like "jvisualvm" during your load test.

Live threads: This shows the current number of live/active threads including both daemon and non-daemon threads(Currently running).

Live Peak: This gives the peak count of live threads since the Java virtual machine started or peak was reset.

Daemon Threads: This gives the current number of live daemon threads.

Total Threads: This gives the total number of threads created and also started since the Java virtual machine started.

So, what am I doing wrong? What am I missing? I have 2 core CPU (max usage during peak hours ~10%) and 2GB of RAM (max usage 60%).CPU usage doesn't grow, RAM is almost half-free if I am having too many requests, shouldn't I have overloaded on the server? Instead, I just get slow response time.

=>IMO, threads are blocking while fetching data from DB. It could be due to the poor performance of query during load times. I would suggest enabling "hibernate.show_sql" capture the SQL. Check the execution plan of the SQL, ensure that indexes are being applied. You can also check the performance of query during load time, by executing it on SQL client.

Answer 4

I'm afraid the question does not currently have enough information for anything other than guesses. The low CPU usage indicates that your java process is waiting for something, which could be anything from obtaining a connection to the database, waiting for the result of a query, or anything else. I would start by looking at what is causing the wait, before trying to fix it. One way to do that is to run

jstack <pid>

(where <pid> would be your process' pid) during peak. It will list a stack trace for each thread. You may be able to spot the problem from that, or you can paste it into the question and perhaps the community can help you out. Good luck with your tuning!

Answer 5

When using spring the default is Tomcat. How ever you can use Netty or Undertow or Jetty for better performance. Please also remember that despite having a 2 core CPU you don't really have 500 threads.

How ever the above answer to actually simulate how your application reacts to traffic is probably the best way to go. If using relation database remember that writes can be even ten times slower than reads (you can see some interesting statistics on that in Cassandra documentation). If using hibernate you may want to look for n+1 problem too. Best way to do that: write an integration test, log sql sent to database. If your test sends 51 instead of one query there you have it.

Answer 6

You can allow as many threads as you want, but if the number of queries increases, then the response time of the RDBMS will deteriorate, which is probably your root cause.

You need to determine where he bottleneck is. Create a dummy page and issue requests to it like a maniac from several computers. If the dummy page responds in time, then your problem is loosely related if at all to connection number and much more to your database. It is highly probable that this is the case.

Take a look at your database, make sure your schema is in normal form. Also, if you search frequently by some columns, make sure you create the correct indexes. Take a look at your queries and observe whether they are unnecessarily slow. If so, optimize them. Cache some data which does not change too frequently and reuse it.