CODEWITHSUNDEEP
pythonpython-3.xmultiprocessingpool
Big-Blue
Big-Blue

Reputation: 429

Issues with Pool and multiprocessing in Python 3

Currently I'm trying to convert my little python script to support multiple threads/cores. I've been reading about the multiprocessing module for several days now and I've also been trying to get it to suit my needs for some time, still I don't have a clue why it won't work.

This is the working code, and this is my approach on implementing the pool workers. As there are no locks in place and I didn't want to make it too complicated at first I already disabled the logging to file.

Still it doesn't work. It doesn't even output any kind of error message. After running it it just displays the welcome message and then it just keeps running, but without outputting any of the desired output, which would be 2 lines per converted file (before + after converting).

Upvotes: 2

Views: 660

Answers (1)

mata
mata

Reputation: 69042

all your workers do is wait for started subprocesses to finish. they don't have any real work to do as that is performed by the external subprocesses, so they will be idle all the time. using multiprocessing for what you do really is overkill, it's much more appropriate to use threads for that.

if you want to learn how to do multiprocessing, try something which involves inter-process communication, synchronisation, pipes, ...

but to also address your question: hava a look at what arguments subprocess.call takes. you call it with a single space-separated command string. if you want that to work you have to pass shell=True, otherwise the whole string is interpreted as the executable's name.
the preferred way to call a program using subprocess is is to specify program and arguments as a list:

subprocess.Popen(['/path/to/program', 'arg1', 'arg2'], *otherarguments)

Upvotes: 1

Related Questions