CODEWITHSUNDEEP
pythonpython-asyncio
JayK23
JayK23

Reputation: 253

How can I run concurrently blocking functions?

I'm creating a Python app where I receive some data and I need to process this data. I can't do this synchronously, since I'm always receiving data, so I need to process the data concurrently.

The problem is that I'm using a library that doesn't support asyncio, so I don't know if I can run it concurrently. Here is my code:

async def event_dispatcher(num):
    data = blocking_operation(num)
    print(data)

async def main():
    tasks = [loop.create_task(event_dispatcher(1)), loop.create_task(event_dispatcher(2)), loop.create_task(event_dispatcher(3)),
             loop.create_task(event_dispatcher(4)), loop.create_task(event_dispatcher(5)), loop.create_task(event_dispatcher(6))]
    
    await asyncio.gather(*tasks)

Where blocking_operation is the function that I cannot run concurrently. So basically I expected the function in tasks to run at the same time, but because of blocking_operation, they will run one at time. Is there any way to solve this? Or am I forced to use another library or multiprocessing?

Upvotes: 0

Views: 1563

Answers (1)

user4815162342
user4815162342

Reputation: 154846

You can use run_in_executor, which will submit the blocking code in a thread pool behind the scenes:

async def event_dispatcher(num):
    loop = asyncio.get_event_loop()
    data = await loop.run_in_executor(None, blocking_operation, num)
    print(data)

Beginning with Python 3.9 you can also use asyncio.to_thread:

async def event_dispatcher(num):
    data = await asyncio.to_thread(blocking_operation, num)
    print(data)

Upvotes: 2

Related Questions