CODEWITHSUNDEEP
pythonpython-asyncio
barciewicz
barciewicz

Reputation: 3773

How to make a function that includes a for loop non-blocking?

I am trying to make the below code asynchronous:

import asyncio
import random

async def count():
    l = []
    for i in range(10000000):
        l.append(i)
    return random.choice(l)

async def long_task1():
    print('Starting task 1...')
    task_output = await count()
    print('Task 1 output is {}'.format(task_output ))


async def long_task2():
    print('Starting task 2...')
    task_output = await count()
    print('Task 2 output is {}'.format(task_output ))

async def main():
    await asyncio.gather(long_task1(), long_task2())

if __name__ == '__main__':
    asyncio.get_event_loop().run_until_complete(main())

Currently it will work synchronously.

Is it because the count function is lacking await statement?

I have tried reworking the function to include await:

async def count():
    l = []
    for i in range(10000000):
        l.append(i)
    choice = await random.choice(l)
    return choice

and it will start asynchronously (both Starting task 1... and Starting task 2... will get printed one after another), but then I get an error:

TypeError: object int can't be used in 'await' expression

I understand that the error happened because the result of random.choice(l) is not an awaitable (a coroutine), but I don't know how to fix this without running in circles. Do I need to somehow refactor the for loop into a couroutine on its own?

Upvotes: 4

Views: 969

Answers (3)

user4815162342
user4815162342

Reputation: 154911

Is it because the count function is lacking await statement?

In short, yes, you have identified the issue correctly. To get parallel execution of tasks, you need not only to specify async def, but also to await something that suspends execution, thus returning control to the event loop. In asyncio that is typically the kind of call that would block a synchronous program, such as a sleep or a read from a socket that is not yet ready for reading.

To force a temporary suspension, you can add await asyncio.sleep(0) inside the loop in count. Adding await in front of an ordinary function such as random.choice, doesn't work because await requires an object that implements the awaitable interface, and in your code random.choice just returns an integer.

Upvotes: 4

balki
balki

Reputation: 27664

For asyncio to work properly, you should not have any cpu intensive task (tight big for loop) in the event loop. As there is no way to get out of the for loop. If you use an explicit asyncio.sleep inside the loop, you are just going in and out of the coroutine unnecessarily and slowing the entire thing. If your goal is just to see how asyncio works, it is fine.

But in real world, if you have a cpu intensive task, you have two options

  1. Use multiprocessing and delegate the task to a different process.
  2. Use a native code binding that releases GIL and uses threads.

As the name suggests, the library is for asynchronous io. async"io"

Upvotes: 1

I Funball
I Funball

Reputation: 380

Your code calls a gather that runs both long_task1 and long_task2 concurrently. Then you call an await on count in each function. But this will await on that subroutine to finish. So the overall subroutine will still finish before the next subroutine begins. You need a function to suspend the entire task. I created two ways to circumvent this. Both involve creating new tasks.

Creating a new subroutine: 

async def count():
   l = []
   await asyncio.wait_for(loopProcess(l), timeout=1000000.0)
   return random.choice(l)

async def loopProcess(l):
   for i in range(10000000):
      l.append(i)

You could also keep your count function the same as the original code and change your long_task(1/2) like this to make count() a new task:

async def long_task1():
   print('Starting task 1...')
   task_output = await asyncio.shield(count())
   print('Task 1 output is {}'.format(task_output ))


async def long_task2():
   print('Starting task 2...')
   task_output = await asyncio.shield(count())
   print('Task 2 output is {}'.format(task_output ))

You can also use create_task if you have python 3.7.

Source: https://docs.python.org/3/library/asyncio-task.html

Upvotes: 1

Related Questions