CODEWITHSUNDEEP
javamultithreading
scm_guy
scm_guy

Reputation: 111

Too many threads running

In our AppServer we are seeing threadcount exceeding its JMX thread count threshold (Max = 500, Actual = 595).

I see there are no deadlocks. I have taken a thread dump and see that out of the 595 threads, majority of them fall in 2 categories:

1) 234 threads in Timed Waiting (Thread Dump shown below)

"Timer-232" daemon prio=10 tid=0x00007f46c85cd000 nid=0x7b06 in Object.wait() [0x00007f4668001000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000075246c498> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x000000075246c498> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

2) 233 threads in Waiting state (Thread Dump shown below). I guess this is the culprit being blocked on a queue, specifically Delay queue. 

"AsyncHttpClient-Reaper" daemon prio=10 tid=0x00007f469cd4c000 nid=0x7b09 waiting on condition [0x00007f4667cfe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007524a2908> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)

Can you please help me understand what is causing the threads to exceed JMX threshold? Its the first time I am working with thread scenario and hence unable to come to a solid analysis. Thanks in advance.

Upvotes: 2

Views: 4967

Answers (2)

Stuart Marks
Stuart Marks

Reputation: 132530

Half of your threads are instances of java.util.TimerThread. An instance of this thread is created whenever something creates an instance of java.util.Timer; every Timer instance creates its own thread.

It looks like the timer thread is in a state where it's waiting for a specific time period to elapse before executing a TimerTask. Since there are so many timer threads, it may be that there is some place in the code that is creating new Timer instances each time it wants to schedule a TimerTask. This will end up creating as many threads as there are pending tasks, which is potentially quite wasteful.

If the tasks to be run are short-lived and are reliable, they can all share a single Timer instance.

However, if a task is long-running, it may delay the execution of subsequent tasks. Or if the task throws an uncaught exception, it will take down the timer thread and prevent subsequent tasks from being executed at all. If either of these is a possibility, you should use Executors.newScheduledThreadPool to create a single ScheduledExecutorService and schedule tasks on it instead of creating new Timer instances.

Upvotes: 2

coffeeaddict
coffeeaddict

Reputation: 868

Looks like you are thrashing your system with too many async requests. Throttle them or set a max pool count for these requests

Upvotes: 2

Related Questions