Java thread dump: BLOCKED thread without "waiting to lock ..."

Question

I'm having difficulties in understanding the thread dump I got from jstack for a Spring MVC web application running on Tomcat 6 (java 1.6.0_22, Linux).

I see blocking threads (that cause other threads to wait) which are blocked themselves, however the thread dump doesn't tell me why or for which monitor they are waiting.

Example:

"TP-Processor75" daemon prio=10 tid=0x00007f3e88448800 nid=0x56f5 waiting for monitor entry [0x00000000472bc000]
    java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Class.initAnnotationsIfNecessary(Class.java:3067)
        - locked <0x00007f3e9a0b3830> (a java.lang.Class for org.catapultframework.resource.ResourceObject)
        at java.lang.Class.getAnnotation(Class.java:3029)
        ...

I.e. I am missing the "waiting to lock ..." line in the stack trace. Apparently the thread locks a Class object, but I don't see why the thread itself is blocked.

The thread-dump does not contain any hints for deadlocks.

What can I do to identify the locking monitor?

Thanks, Oliver

Answer 1

Check if the finalizer thread is blocked or waiting.

During a GC sweep, the GC will "stop the world" to perform its cleanup. The definition of "world" depends on the garbage collector being used and context. It may be a small cluster of threads or all of them. Before officially collecting garbage, GC will invoke the object's finalize().

If you are in the undesirable situation where you are implementing finalizer methods, the finalization code may be blocking it from finishing and the 'world' stays stopped.

This is most obvious when seeing lots of threads being permanently-blocked by some unknown magic force: Look up the code where the blocking occurs and it will make no sense; there is no blocking code to be found anywhere near it and the dumps will not divulge what monitor it is waiting on because there isn't one. The GC has paused the threads.

Answer 2

This is a cosmetic bug in Oracle's HotSpot JVM - in your stack trace where you see - locked <0x00007f3e9a0b3830> it should actually say - waiting to lock <0x00007f3e9a0b3830>.

See this bug for more details.

Answer 3

I had a similar problem just now using an Applet in Google Chrome.

In short:

• The BLOCKED threads can be blocked when the VM needs to load a class.
• When the process of loading the class itself is blocked by something a freeze for the whole app can occur.

In Detail:

I had the following scenario:

1. I am using an Applet in Chrome with codebase = folder to single class-files (no jar)
2. The Website passes focus-events to the applet using LiveConnect
3. The incoming JS-calls are using an Executor with new Runnable() ... to detach the calls in order to reduce the wait times and thus hangs in JS.
4. That's where the problem occured!

Explanation:

• The new Runnable() is an annonymous inner class which was not loaded before the JS call happened.
• The JS call therefore triggers the class load.
• But now the class loader is blocked because it needs to talk to the browser (i am guessing) via the same queue or mechanism that is processing the incoming JS call.

Here is the blocked thread that is trying to load the class:

"Thread-20" daemon prio=4 tid=0x052e8400 nid=0x4608 in Object.wait() [0x0975d000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at sun.plugin2.message.Queue.waitForMessage(Unknown Source)
    - locked <0x29fbc5d8> (a sun.plugin2.message.Queue)
    at sun.plugin2.message.Pipe$2.run(Unknown Source)
    at com.sun.deploy.util.Waiter$1.wait(Unknown Source)
    at com.sun.deploy.util.Waiter.runAndWait(Unknown Source)
    at sun.plugin2.message.Pipe.receive(Unknown Source)
    at sun.plugin2.main.client.MessagePassingExecutionContext.doCookieOp(Unknown Source)
    at sun.plugin2.main.client.MessagePassingExecutionContext.getCookie(Unknown Source)
    at sun.plugin2.main.client.PluginCookieSelector.getCookieFromBrowser(Unknown Source)
    at com.sun.deploy.net.cookie.DeployCookieSelector.getCookieInfo(Unknown Source)
    at com.sun.deploy.net.cookie.DeployCookieSelector.get(Unknown Source)
    - locked <0x298da868> (a sun.plugin2.main.client.PluginCookieSelector)
    at sun.net.www.protocol.http.HttpURLConnection.setCookieHeader(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    - locked <0x2457cdc0> (a sun.net.www.protocol.http.HttpURLConnection)
    at com.sun.deploy.net.HttpUtils.followRedirects(Unknown Source)
    at com.sun.deploy.net.BasicHttpRequest.doRequest(Unknown Source)
    at com.sun.deploy.net.BasicHttpRequest.doGetRequestEX(Unknown Source)
    at com.sun.deploy.cache.ResourceProviderImpl.checkUpdateAvailable(Unknown Source)
    at com.sun.deploy.cache.ResourceProviderImpl.isUpdateAvailable(Unknown Source)
    at com.sun.deploy.cache.DeployCacheHandler.get(Unknown Source)
    - locked <0x245727a0> (a java.lang.Object)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    - locked <0x24572020> (a sun.net.www.protocol.http.HttpURLConnection)
    at java.net.HttpURLConnection.getResponseCode(Unknown Source)
    at sun.plugin2.applet.Applet2ClassLoader.getBytes(Unknown Source)
    at sun.plugin2.applet.Applet2ClassLoader.access$000(Unknown Source)
    at sun.plugin2.applet.Applet2ClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.plugin2.applet.Applet2ClassLoader.findClass(Unknown Source)
    at sun.plugin2.applet.Plugin2ClassLoader.loadClass0(Unknown Source)
    at sun.plugin2.applet.Plugin2ClassLoader.loadClass(Unknown Source)
    - locked <0x299726b8> (a sun.plugin2.applet.Applet2ClassLoader)
    at sun.plugin2.applet.Plugin2ClassLoader.loadClass(Unknown Source)
    - locked <0x299726b8> (a sun.plugin2.applet.Applet2ClassLoader)
    at java.lang.ClassLoader.loadClass(Unknown Source)

As you can see it is waiting for a message --> waitForMessage().

At the same time there is our incoming JS call being BLOCKED here:

"Applet 1 LiveConnect Worker Thread" prio=4 tid=0x05231800 nid=0x1278 waiting for monitor entry [0x0770e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at MyClass.myMethod(MyClass.java:23)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at sun.plugin.javascript.Trampoline.invoke(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at sun.plugin.javascript.JSClassLoader.invoke(Unknown Source)
    at sun.plugin2.liveconnect.JavaClass$MethodInfo.invoke(Unknown Source)
    at sun.plugin2.liveconnect.JavaClass$MemberBundle.invoke(Unknown Source)
    at sun.plugin2.liveconnect.JavaClass.invoke0(Unknown Source)
    at sun.plugin2.liveconnect.JavaClass.invoke(Unknown Source)
    at sun.plugin2.main.client.LiveConnectSupport$PerAppletInfo$DefaultInvocationDelegate.invoke(Unknown Source)
    at sun.plugin2.main.client.LiveConnectSupport$PerAppletInfo$3.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.plugin2.main.client.LiveConnectSupport$PerAppletInfo.doObjectOp(Unknown Source)
    at sun.plugin2.main.client.LiveConnectSupport$PerAppletInfo$LiveConnectWorker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

Additional other threads were blocked in the same manner. I suppose all subsequent class-load requests were blocked by the first blocked class-loading thread.

As mentioned before, my guess is that the class-loading process is blocked by the pending JS call, which by itself is blocked by the missing class to be loaded.

Solutions:

1. Trigger loading all relevant classes in the constructor of the applet before any calls can be made from JS.
2. It might help if the class-files are not being loaded individually, but from a jar-file. The theory behind this is: The class loader does not need to talk to the browser to load the classes from the jar-file (which would be
3. In combination with 1.: Use a dynamic Proxy class to wrap all incoming JS calls and run them independently in an Executor.

My implementation for #3:

public class MyClass implements JsCallInterface
{

    private final JsCallInterface jsProxy;

    private final static interface JsCallInterface
    {
        public void myMethod1Intern(String param1, String param2);
    }

    private final class JsCallRunnable implements Runnable
    {

        private final Method method;
        private final Object[] args;

        private JsCallRunnable(Method method, Object[] args)
        {
            this.method = method;
            this.args = args;
        }

        public void run()
        {
            try
            {
                method.invoke(MyClass.this, args);
            }
            catch (Exception e)
            {
                e.printStackTrace();
            }
        }
    }

    public MyClass()
    {
        MyUtilsClass.class.getName(); // load class
        JsCallRunnable.class.getName(); // load class
        InvocationHandler jsCallHandler = new InvocationHandler()
        {

            public Object invoke(final Object proxy, final Method method, final Object[] args) throws Throwable
            {
                MyUtilsClass.executeInExecutor(new JsCallRunnable(method, args));
                return null;
            }

        };
        jsProxy = (JsCallInterface) Proxy.newProxyInstance(MyClass.class.getClassLoader(), new Class<?>[] { JsCallInterface.class }, jsCallHandler);
    }

    public void myMethod1(String param1, String param2)
    {
        jsProxy.myMethod1Intern(param1, param2);
        // needs to be named differently than the external method or else the proxy will call this method recursively
        // alternatively the target-class in "method.invoke(MyClass.this, args);" could be a different instance of JsCallInterface
    }

    public void myMethod1Intern(String param1, String param2)
    {
        // do actual work here
    }
}

Answer 4

Apparently the situation where we observed these kinds of blocked threads were related to heavy memory consumption and therefore massive garbage collection.

This question Java blocking issue: Why would JVM block threads in many different classes/methods? describes a similar situation, so I believe these threads were simply blocked by the garbage collector.

(Anyway, after solving the memory issue this problem with the blocking threads was gone.)