CODEWITHSUNDEEP
python-3.xconcurrent.futures
qweruiop
qweruiop

Reputation: 3266

Is `concurrent.futures.map` thread-safe?

I find it not mentioned by the doc: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor.map

Upvotes: 3

Views: 2459

Answers (2)

Queequeg
Queequeg

Reputation: 1

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

resultDict = {0: [], 1: [], 2: [], 3: [], 4: []}
resultDict1 = {0: [], 1: [], 2: [], 3: [], 4: []}

def appendResult(a, b):
  result = a ** b
  resultDict[result % 5].append(result)
  time.sleep(1)

def appendResult1(a, b):
  result = a ** b
  resultDict1[result % 5].append(result)
  time.sleep(1)

startTime = time.time()
processes = []
with ThreadPoolExecutor(max_workers=12) as executor:
  for i in range(100):
    processes.append(executor.submit(appendResult, i, 2))

for task in as_completed(processes):
  task.result()

print("Cost", time.time() - startTime, "s")


startTime = time.time()
for i in range(100):
  appendResult1(i, 2)
print("Cost", time.time() - startTime, "s")

Yes, it's thread-safe. So resultDict and resultDict1 will be the same at the end of the above code.

Upvotes: 0

Sun Bear
Sun Bear

Reputation: 8254

Reference:

According to ThreadPoolExecutor

Deadlocks can occur when the callable associated with a Future waits on the results of another Future.

The 2 examples there shows how deadlock can happen. Try replacing .submit() with .map() and make the other necessary changes.`

Under-the-hood:

According to Python's python3.6/concurrent/futures/thread.py module (search your system for this file, the class ThreadPoolExecutor actually uses queue.Queue() (see line 107) to implement python threading and use the primitive threading.Lock() (see line 110) to lock the threads.

Explanation:

If to you "thread-safe" means multiple threads in a program each attempting to access a common data structure or location in memory, then you should know that concurrent.futures.ThreadPoolExecutor allow only one thread to access the common data structure or location in memory at a time; the threading.Lock() primitive is used to manage this. And when a function in one of your thread needs to wait for the results in another thread, then deadlock can occur and your code won't work; this you should avoid.

Upvotes: 5

Related Questions