Java thread dump CPU or just IO wait?

Question

I have copied a suspicious thread dump. I see 173 threads in IN_NATIVE state on call java.net.SocketInputStream.socketRead0() and 30 threads in BLOCKED state on the same call.
I think BLOCKED state doesn't consume CPU, but they are waiting on IO.

What about threads in IN_NATIVE state? do they consume CPU? Any idea, why am I seeing different thread states for the same call socketRead0() ?

173 threads
Thread 40283 - threadId:Thread 40283 - state:IN_NATIVE
stackTrace:
java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
java.net.SocketInputStream.read(byte[], int, int, int) @bci=87, line=152 (Compiled frame)
java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=122 (Compiled frame)
com.sun.mail.util.TraceInputStream.read(byte[], int, int) @bci=7, line=110 (Compiled frame)
java.io.BufferedInputStream.fill() @bci=175, line=235 (Compiled frame)
java.io.BufferedInputStream.read() @bci=12, line=254 (Compiled frame)
com.sun.mail.util.LineInputStream.readLine() @bci=33, line=88 (Compiled frame)
com.sun.mail.smtp.SMTPTransport.readServerResponse() @bci=43, line=1589 (Compiled frame)
com.sun.mail.smtp.SMTPTransport.openServer(java.lang.String, int) @bci=117, line=1369 (Compiled frame)
com.sun.mail.smtp.SMTPTransport.protocolConnect(java.lang.String, int, java.lang.String, java.lang.String) @bci=270, line=412 (Compiled frame)
javax.mail.Service.connect(java.lang.String, int, java.lang.String, java.lang.String) @bci=380, line=288 (Compiled frame)
com.healthies.push.injectors.mailsenders.DefaultMailSender.sendMessage(com.healthies.push.messages.EmailMessage, javax.mail.internet.MimeMessage) @bci=28, line=63 (Compiled frame)
com.healthies.push.injectors.SMTPMessageInjector.sendMessage(com.healthies.push.injectors.mailsenders.MailSender, com.healthies.push.messages.EmailMessage, javax.mail.internet.MimeMessage) @bci=55, line=140 (Compiled frame)
com.healthies.push.injectors.SMTPMessageInjector.inject(com.healthies.push.messages.EmailMessage) @bci=210, line=117 (Compiled frame)
com.healthies.push.injectors.SMTPMessageInjector.inject(com.healthies.push.messages.Message) @bci=5, line=35 (Compiled frame)
com.healthies.push.injectors.EmailInjectorPool$1.run() @bci=85, line=40 (Compiled frame)
java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Compiled frame)
java.util.concurrent.FutureTask.run() @bci=42, line=262 (Compiled frame)
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Compiled frame)
java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
--------------------------------------------------------------------
30 threads 
Thread 40276 - threadId:Thread 40276 - state:BLOCKED
stackTrace:
java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
java.net.SocketInputStream.read(byte[], int, int, int) @bci=87, line=152 (Compiled frame)
java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=122 (Compiled frame)
com.sun.mail.util.TraceInputStream.read(byte[], int, int) @bci=7, line=110 (Compiled frame)
java.io.BufferedInputStream.fill() @bci=175, line=235 (Compiled frame)
java.io.BufferedInputStream.read() @bci=12, line=254 (Compiled frame)
com.sun.mail.util.LineInputStream.readLine() @bci=33, line=88 (Compiled frame)
com.sun.mail.smtp.SMTPTransport.readServerResponse() @bci=43, line=1589 (Compiled frame)
com.sun.mail.smtp.SMTPTransport.issueSendCommand(java.lang.String, int) @bci=6, line=1494 (Compiled frame)
com.sun.mail.smtp.SMTPTransport.finishData() @bci=34, line=1321 (Compiled frame)
com.sun.mail.smtp.SMTPTransport.sendMessage(javax.mail.Message, javax.mail.Address[]) @bci=281, line=637 (Compiled frame)
com.healthies.push.injectors.mailsenders.DefaultMailSender.sendMessage(com.healthies.push.messages.EmailMessage, javax.mail.internet.MimeMessage) @bci=37, line=64 (Compiled frame)
com.healthies.push.injectors.SMTPMessageInjector.sendMessage(com.healthies.push.injectors.mailsenders.MailSender, com.healthies.push.messages.EmailMessage, javax.mail.internet.MimeMessage) @bci=55, line=140 (Compiled frame)
com.healthies.push.injectors.SMTPMessageInjector.inject(com.healthies.push.messages.EmailMessage) @bci=210, line=117 (Compiled frame)
com.healthies.push.injectors.SMTPMessageInjector.inject(com.healthies.push.messages.Message) @bci=5, line=35 (Compiled frame)
com.healthies.push.injectors.EmailInjectorPool$1.run() @bci=85, line=40 (Compiled frame)
java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Compiled frame)
java.util.concurrent.FutureTask.run() @bci=42, line=262 (Compiled frame)
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Compiled frame)
java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Answer 1

Normally you will never see a thread executing non-synchronized native method in BLOCKED state. But this dump is obtained in forced mode (jstack -F), which works differently.

It seems like JVM has requested a safepoint operation. All threads running Java code must be stopped at safepoint, but native methods may still run. Whenever a native method returns, it checks whether a safepoint operation is in progress, and if it is, blocks current thread until the VM operation completes.

So, in your case

• threads in BLOCKED state have already completed the native call and are waiting for the end of safepoint operation at the return from the method;
• threads in IN_NATIVE state are still executing a native call which may be blocked on a socket read.

JVM has no means to distinguish a native method blocked on a system call from a native method actually consuming CPU - in both cases it will show a thread in IN_NATIVE state. You cannot conclude from a thread dump whether those threads are busy or not.

In order to see whether a native method consumes CPU, you will need a utility like top, perf or a native code aware profiler.