CODEWITHSUNDEEP
python-2.7multiprocessingexternal-processcprofile
erik
erik

Reputation: 3880

Profiling a Python program that calls parallel external programs

Similar to this question on profiling multiprocessing code...

I need to profile my codebase, and I'd planned to use cProfile. This program is essentially a genetic algorithm that runs an evaluation function in parallel. The kicker is that this evaluation function externally calls another Python program via the command line.

Is it possible to use cProfile to profile the overall run? Basically, I'd intended to start the profiling inside my worker function (cumulative time, pcalls, memory overhead, etc.), however I'm concerned that the external programs will not be visible to the profiler.

Upvotes: 1

Views: 1282

Answers (1)

skrrgwasme
skrrgwasme

Reputation: 9633

You are correct that the external programs will not be "visible" to the profiler. The cProfile module adds software hooks to your code that trigger various counters on events like function calls or returns. You will be able to see your call to subprocess.call() or whatever you're using to execute the external commands, but you will not see any detail about the external command's execution. You will be able to see how long the call blocked, so you'll get an idea of how long the external command takes to execute, but you won't get the profiling details of them.

That being said, there's no reason you can't add the cProfile module to the called script as well, and you'll get profile results for both your main script and the called external script. They just won't be in the same file.

Upvotes: 1

Related Questions