How to get thread debug information from running production Tomcat web application?

Question

We have a production web application that appears to be having a thread stuck doing a bunch of work, and running up the CPU and load average. I know pretty much what the issue is - our DWR service is converting something extremely large to a Javascript object response. The problem is I don't know which service call is producing this large amount of data, because the thread is outside any of my code (it is in the DWR servlet converting my return value to the outbound javascript object... otherwise I could see my service method in the thread dump).

You don't need to know about DWR to help me though, because my question is this:

Is this any way for me to look into a deployed web application (Tomcat 6) and see what kind of state a thread is in, such as variable values, etc.? It would be perfect if I could take basically a "debug snapshot" of the thread, as if I had a breakpoint in debug mode during local development, but any solution that let's me see into or print out the value of something would be useful.

Thanks!

UPDATE

If it is possible to not restart the server, that would be best, because this issue only comes up every once in awhile, and restarting makes the issue go away. Looks like JProfiler requires adding the -agentpath option to the JVM, so I would need to restart it.

Answer 1

You could also send a SIGQUIT signal (numeric code 3) to the java process of your Tomcat instance, e.g. by doing kill -3 <pid>. This makes java dump the stacks of all threads to sysout -- it does NOT shut down the process, even though the name might imply that.

Answer 2

You can open the source in eclipse and attach a remote debugger. Its really no different from any other Java application. Remote Debugging Article. You can insert breakpoints in the app and pause the threads at will.

Another option is a profiler like JProfiler, Yourkit, or VisualVM. This tools allow you to do thread dumps where you can investigate the state of all running threads and their monitors. Beyond that you can watch the threads go in real time, see how much time their active when and how long they are sleeping and correlate that info with the thread dumps to get an idea of where your application is spending its time. What might be hard to get with a profiler is the state of individual variables.

Answer 3

Yes, you can attach Visual VM 1.3.2, with all the plugins downloaded and installed, to see the state of every thread, the generations in the heap, etc.