CODEWITHSUNDEEP
pythonmutexconcurrent.futures
xorsyst
xorsyst

Reputation: 8257

Using mutexes with concurrent futures in Python

I have some code that uses concurrent futures to connect to lots of remote hosts to run some commands.

For example:

def set_host_to(host, value):
  connection = connect_to(host)
  info = do_something_with(connection)
  do_some_action(connection, value)

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
  for host, value in things_to_do:
    executor.submit(set_host_to, host, value)

I've now got a requirement that some of those futures don't run at the same time, but which ones can't be detemined until they've got the info above.

Is the right solution some sort of mutexes here? Say info (or some part of it) was a string, and 2 futures with the same string shouldn't run at the same time. How would I go about coding that?

Upvotes: 2

Views: 2029

Answers (1)

xorsyst
xorsyst

Reputation: 8257

This can be done with a dictionary of locks, using the code below.

locks = defaultdict(threading.Lock)
main_lock = threading.Lock() 

with main_lock:
    lock = locks[info]
with lock:
    do_some_action(connection, value)

I'm not sure if the main lock is necessary, but this seems to work.

Upvotes: 2

Related Questions