CODEWITHSUNDEEP
pythonmultithreadingpython-multithreading
PoweredByCoffee
PoweredByCoffee

Reputation: 1193

Python Multiprocessing Causing Entire Script to Loop

It seems multiprocessing swaps between threads faster so I started working on swapping over but I'm getting some unexpected results. It causes my entire script to loop several times when a thread didn't before.

Snippet example:

stuff_needs_done = true
more_stuff_needs_done =  true
print "Doing stuff"

def ThreadStuff():
  while 1 == 1:
    #do stuff here

def OtherThreadStuff():
  while 1 == 1:
    #do other stuff here

if stuff_needs_done == true:
  Thread(target=ThreadStuff).start()

if more_stuff_needs_done == true:
  Thread(target=OtherThreadStuff).start()

This works as I'd expect. The threads start and run until stopped. But when running a lot of these the overhead is higher (so I'm told) so I tried swapping to multiprocessing.

Snippet example:

stuff_needs_done = true
more_stuff_needs_done = true
print "Doing stuff"


def ThreadStuff():
  while 1 == 1:
    #do stuff here

def OtherThreadStuff():
  while 1 == 1:
    #do other stuff here

if stuff_needs_done == true:
  stuffproc1= Process(target=ThreadStuff).start()

if more_stuff_needs_done == true:
  stuffproc1= Process(target=OtherThreadStuff).start()

But what seems to happen is the whole thing starts a couple of times so the "Doing stuff" output comes up and a couple of the threads run.

I could put some .join()s in but there is no loop which should cause the print output to run again which means there is nowhere for it to wait.

My hope is this is just a syntax thing but I'm stumped trying to find out why the whole script loops. I'd really appreciate any pointers in the right direction.

Upvotes: 0

Views: 476

Answers (1)

Francisco
Francisco

Reputation: 11496

This is mentioned in the docs:

Safe importing of main module

Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).

For example, under Windows running the following module would fail with a RuntimeError:

from multiprocessing import Process

def foo():
    print 'hello'

p = Process(target=foo)
p.start()

Instead one should protect the “entry point” of the program by using if __name__ == '__main__': as follows:

from multiprocessing import Process, freeze_support

def foo():
    print 'hello'

if __name__ == '__main__':
    freeze_support()
    p = Process(target=foo)
    p.start()

This allows the newly spawned Python interpreter to safely import the module and then run the module’s foo() function.

Upvotes: 1

Related Questions