Profile Python CPU Usage By Thread

Question

I've got a multi-threaded Python application, and I'm currently troubleshooting very high (90% or more) CPU usage.

I'm going to be trying out the profiler, but I wanted to see if there is a way I can get CPU usage per thread from within the application. I understand that os.times() will get CPU usage overall - is there something I can run from within each thread to get each thread's usage? It would be very helpful to determine which thread is eating CPU.

Answer 1

Or you can simply use yappi. (https://code.google.com/p/yappi/) It transparently uses GetThreadTimes() if CPU clock type is selected for profiling. It will show you CPU time of a running thread.

See here also for an example: https://code.google.com/p/yappi/wiki/YThreadStats_v082

Answer 2

If you're on Windows, you could take a look at the Process Explorer ("procexp.exe" shipped with Microsoft's Sysinternals Tools). There's a tab "Threads" in the properties of each process which lists all threads and their CPU time.

If that doesn't give you enough information, use Python's profiling mechanisms, which aren't too hard to use.

Answer 3

Other than the profiler, I don't think Python has much built-in for monitoring individual thread CPU usage. Also, this post provides an example of a simple implementation of a thread profiler. It appears mainly geared towards Unix-based systems, but it's a start.