Reputation: 15678
I have a python application that spawns a separate process to do some work (I ran into performance issues using threads due to the GIL (global interpreter lock)). Now what's my methods in python to synchronize shared resources across processes?
I move data into a queue and a spawn process does the work when as it receives data from that queue. But I need to be able to guarantee that the data is coming out in an orderly fashion, same order as it was copied in so I need to guarantee that only one process at any time can read/write from/to the queue. How do I do that best?
Thanks, Ron
Upvotes: 3
Views: 6760
Reputation: 7906
I think you need a Semaphore, check this example code:
import threading
import datetime
class ThreadClass(threading.Thread):
def run(self):
now = datetime.datetime.now()
pool.acquire()
print "%s says hello, World! at time: %s" % (self.getName(),now)
pool.release()
pool = threading.BoundedSemaphore(value=1)
for i in range(10):
t = ThreadClass()
t.start()
Has this output:
Thread-1 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-2 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-3 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-4 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-5 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-6 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-7 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-8 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-9 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-10 says hello, World! at time: 2013-05-20 18:57:47.609000
Where as:
import threading
import datetime
class ThreadClass(threading.Thread):
def run(self):
now = datetime.datetime.now()
print "%s says hello, World! at time: %s" % (self.getName(),now)
for i in range(10):
t = ThreadClass()
t.start()
Has this output:
Thread-1 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-2 says hello, World! at time: 2013-05-20 18:58:05.
531000
Thread-4 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-3 says hello, World! at time: 2013-05-20 18:58:05
.531000
Thread-6 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-5 says hello, World! at time: 2013-05-20 18:58:05
.531000
Thread-8 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-7 says hello, World! at time: 2013-05-20 18:58:05
.531000
Thread-10 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-9 says hello, World! at time: 2013-05-20 18:58:0
5.531000
So with a BoundedSemaphore in python you can make sure that before anyone writes to your queue they have to have the semaphore. This doesn't ensure your results are added to the queue in the correct order though.
EDIT:
If you going to do this and keep order your going to need something like this:
import multiprocessing
import datetime
import random
import time
def funfun(number):
time.sleep(random.randint(0,10))
now = datetime.datetime.now()
return "%s says hello, World! at time: %s" % (number,now)
if __name__ == "__main__":
pool = multiprocessing.Pool(10)
for item in pool.imap(funfun,[i for i in range(10)]):
print item
which prints:
0 says hello, World! at time: 2013-05-21 00:38:48.546000
1 says hello, World! at time: 2013-05-21 00:38:55.562000
2 says hello, World! at time: 2013-05-21 00:38:47.562000
3 says hello, World! at time: 2013-05-21 00:38:51.578000
4 says hello, World! at time: 2013-05-21 00:38:50.578000
5 says hello, World! at time: 2013-05-21 00:38:48.593000
6 says hello, World! at time: 2013-05-21 00:38:52.593000
7 says hello, World! at time: 2013-05-21 00:38:48.593000
8 says hello, World! at time: 2013-05-21 00:38:50.593000
9 says hello, World! at time: 2013-05-21 00:38:51.609000
So with this you could just append to the queue in the right order, and the jobs will wait for their turn to add to the queue.
Upvotes: 2