CODEWITHSUNDEEP
pythonpython-asyncio
InfoLearner
InfoLearner

Reputation: 15598

Python thread safe asyncio

I am trying to run my sync functions using asyncio.run_in_executor

My coroutines shares data and update data. The run in executor starts threads. Do I need to make my objects thread and asyncio safe?

import asyncio
from concurrent.futures import ThreadPoolExecutor

class shared:
  #complex object

def func(a, b):
    # blocking calls
    return a + b

async def main(loop):
  
    result = await loop.run_in_executor(None, func, "Hello,", " world!")
    UpdateSharedObject(result) 

Start main as 5 tasks using create_task

Upvotes: 1

Views: 2373

Answers (2)

Ilya Egorov
Ilya Egorov

Reputation: 251

Since you mentioned in your question comment that the blocking call is I/O bound in nature, I can assume that you may need to be in a critical section for a long time. In this case synchronization primitives from the threading module will not suit you: they will block the event loop. Instead, I suggest using primitives from my aiologic package, such as aiologic.Lock:

lock = aiologic.Lock()

with lock:
    # change the state of a complex object (in a worker thread)

async with lock:
    # change the state of a complex object (in an async task)

There are other kinds of synchronization primitives in aiologic. But if all you need is to make your object thread-safe, you only need to know what primitives can be used as a lock.

Upvotes: 0

alex_noname
alex_noname

Reputation: 32153

If you are accessing (and especially making changes) to the same resource from different threads. Then I highly recommend using synchronization primitives. Because Python does not provide this thread-safety in all cases. Many Python operations are atomic, but not all. Actually, single opcodes are thread-safe.

Examples of thread-safe operations:

L.append(x)
L1.extend(L2)
x = L[i]
x = L.pop()

These are not:

i = i+1
L.append(L[-1])
L[i] = L[j]
D[x] = D[x] + 1

More info here: https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe

As for asyncio, I agree with the @user4815162342 in the comments and advise you to redesign the program so that the functions executed in the threads are pure functions and do not change shared objects. Because the use of thread synchronization, together with cooperative multitasking (asyncio), can greatly complicate the code and lead to subtle bugs.

Upvotes: 1

Related Questions