CODEWITHSUNDEEP
pythondebuggingdeadlockmultiprocessing
Fragsworth
Fragsworth

Reputation: 35497

Multiprocessing debug techniques

I'm having trouble debugging a multi-process application (specifically using a process pool in python's multiprocessing module). I have an apparent deadlock and I do not know what is causing it. The stack trace is not sufficient to describe the issue, as it only displays code in the multiprocessing module.

Are there any python tools, or otherwise general techniques used to debug deadlocks?

Upvotes: 29

Views: 22522

Answers (2)

brool
brool

Reputation: 2135

Yah, debugging deadlocks is fun. You can set the logging level to be higher -- see the Python documentation for a description of it, but really quickly:

import multiprocessing, logging
logger = multiprocessing.log_to_stderr()
logger.setLevel(multiprocessing.SUBDEBUG)

Also, add logging for anything in your code that deals with a resource or whatnot that might be in contention. Finally, shot in the dark: spawning off child processes during an import might cause a problem.

Upvotes: 46

Eric O. Lebigot
Eric O. Lebigot

Reputation: 94485

In order to avoid deadlocks in the first place, learning good practices is useful, as parallel processing is indeed quite subtle. The (free) Little Book of Semaphores can be a very enjoyable read!

Upvotes: 12

Related Questions