CODEWITHSUNDEEP
javaperformance
Timo Ernst
Timo Ernst

Reputation: 15993

How to track down the cause for performance loss?

I am running a Java-based, multi-user web application on Tomcat 6.

For some reason, after 1 day the whole application gets really slow.

"top" says that CPU is at 120% (caused by the tomcat process) although there are not many users online, so I assume there is some kind of infinite loop (or a similar issue) going on.

On my local system, I cannot reproduce the problem. I also cannot find out which part (which class/method) is causing the problem.

What's the best way to debug this?

Upvotes: 2

Views: 895

Answers (6)

deeKay
deeKay

Reputation: 287

If you cannot reproduce it on local system, chances are it is related to threads.

I would suggest to you to follow these steps.

  1. Take a thread dump with jstack. (jstack -l )
  2. Check out in the beginning of the dump file if it says something about deadlock.

If you don't find anything, take another dump. (Keep the first one) 1. What all thread you see looks like stuck in the same position as compared to thread dump one. (Ignore the one who are waiting for job from acceptor thread. Specific to Tomcat)

Similar issues that I have faced in the past are

  1. Hashmap get into infinite loop. It will spin your CPU to 100%.
  2. Some loop running infinite because of certain condition not being met.
  3. Garbage collection spinning the cpu. You can detect if gc is running too by either putting on extra params to JVM.

Good Luck.

Upvotes: 0

jmaeder
jmaeder

Reputation: 21

If you have the possibility to stop the application server:

  • instrument your application with a java agent (see [1] for more information). Some products are using this technique to give you J2EE performance metrics (Average response time, response per interval, concurrent invocations, aso.) out of the box (Wily Introscope) Thus, you can identify quickly the layer/location of the problem (frontend, backend, etc);

  • use a Java monitoring console (like visualVM), to monitor the GC activity. All you need id to enable JMX; For your specific problem, it is probable that the GC is consuming a lot of CPU cycles to garbage the memory;

If you don't have the possibility to stop the application:

  • Parse your log file searching for performance metrics
  • Parse specific logfiles (f.exemple verbose GC log files)

[1] http://download.oracle.com/javase/6/docs/api/java/lang/instrument/package-summary.html

Upvotes: 0

Matteo
Matteo

Reputation: 14940

Use a profiler and see which methods are running the most (most calls or most of the time). jvisualvm is free and part of Java JDK.

If a GUI is not an option (as stated in the comment) try with hprof (http://java.sun.com/developer/technicalArticles/Programming/HPROF.html) or any command line profiler (there are many of them).

But you don't need to run the profiler GUI on the same machine. You could run it on your machine and connect to the remote VM.

Upvotes: 8

developer
developer

Reputation: 9478

there are many profilers to know which thread consuming more time of jvm and also we can have suggestion for those from the tool itself.below are some profilers

Jprofiler, tptpprofiler

Upvotes: 0

mohdajami
mohdajami

Reputation: 9690

Happened to me once, and this was my plan to deal with it:

  1. Go to the Production environment or whatever env the problem happens on and work from there all day.
  2. Try to focus you work on specific parts/modules of the application every one or two hours.
  3. Look for any exceptions in the log files.
  4. After doing all this you might get to know exactly or at least roughly what part of the system is causing the problem.

Good luck

Upvotes: 0

Thomas
Thomas

Reputation: 88747

I don't know tomcat itself (we're using JBoss which has tomcat integrated) - but a first step would be to create a thread dump and see which thread is doing what.

If your application is running on Java 6, you might use jConsole to connect to the JVM and create the thread dump. Note that remote access to the JVM might still have to be enabled.

Upvotes: 3

Related Questions