CODEWITHSUNDEEP
windowsmultithreadingpython-2.7multiprocessingpyserial
eecs
eecs

Reputation: 141

Python Multiprocessing slight slower than Multithreading on Windows

I have been experimenting my code to send "parallel" commands to multiple serial COM ports.

My multi-threading code consists of:

global q 
q = Queue()
devices = [0, 1, 2, 3]
for i in devices:
    q.put(i)
cpus=cpu_count() #detect number of cores
logging.debug("Creating %d threads" % cpus)
for i in range(cpus):
    t = Thread(name= 'DeviceThread_'+str(i), target=testFunc1)
    t.daemon = True
    t.start()

and multi-processing code consists of:

devices = [0, 1, 2, 3]
cpus=cpu_count() #detect number of cores
pool = Pool(cpus)
results = pool.map(multi_run_wrapper, devices)

I observe that the task of sending serial commands to 4 COM ports in "parallel" takes about 6 seconds and multi-processing always always takes a 0.5 to 1 second of additional total run time.

Any inputs on why the discrepancy on a Windows machine?

Upvotes: 0

Views: 417

Answers (1)

ShadowRanger
ShadowRanger

Reputation: 155418

Well, for one, you're not comparing apples to apples. If you want equivalent code, use multiprocessing.dummy.Pool in your threaded case (which is the same as multiprocessing.Pool implemented in terms of threads, not processes), so you're at least using the same basic parallelization model with different internal implementations, not changing everything all at once.

Beyond that, launching the workers and communicating data to them has some overhead, more on Windows than on other systems since Windows can't fork to spawn new processes cheaply; it has to spawn a new Python instance and then copy over state via IPC to approximate forking.

Aside from that, you haven't provided enough information; your process and thread based worker functions aren't provided, and could cause significant differences in behavior. Nor have you provided information on how you're performing timing. Similarly, if each worker process needs to reinitialize the COM port communication library, that could involve non-trivial overhead.

Upvotes: 1

Related Questions