CODEWITHSUNDEEP
pythonpython-asyncio
user3553653
user3553653

Reputation: 127

Making asyncio gather wait until all tasks are done

I have functions that are supposed to run in paraller and gather data for a second, and then return the data once the time has passed.

Problem is, one task (to my understanding) finishes before the other, causing results to be empty on that result. Below code for example:

import asyncio
import time

async def task(id, end_time):
    print('Started task ', id)
    results = []
    while time.time() < end_time:
        results.append(1)
    return results


async def main():
    while True:
        end_time = time.time() + 1
        results = await asyncio.gather(*[task(i,end_time) for i in range(2)])
        print(len(results[0]), len(results[1]))
try:
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
except KeyboardInterrupt:
    pass
finally:
    loop.close()

returns

Started task  0
Started task  1
5580452 0

Where am I going wrong? I know I should somehow make it wait for all tasks to be finished before doing things with the results, but don't really know how to achieve this, after a while of fiddling around with different things.

Upvotes: 2

Views: 7623

Answers (1)

Tom Wojcik
Tom Wojcik

Reputation: 6179

You don't need to gather them differently. The way you use asyncio is ok.

In your task coroutine remove the while and just append to results once. You will see that both tasks will return their outputs.

The problem is that when you do while time.time() you lock it, as it's not asynchronous. So it runs the while loop until the end_time condition is met.

Then the second one starts, since the task coro never allowed anything else to be executed in the meantime. As both were using the same end_time, the second coro will never append.

If you want to run the while loop asynchronously, you need to create additional asynchronous logic for it and await it in your task function.

EDIT: For example, if you just want put it back to the event loop and allow something else to be executed, simply put await asyncio.sleep(0.1) in your while loop.

Example:

async def task(id, end_time):
    print('Started task ', id)
    results = []
    while time.time() < end_time:
        results.append(1)
        await asyncio.sleep(0.000000001)
    return results

Room for improvement:

  • asyncio.run(main() takes care of creating event loop
  • appending to list is an expensive operation, try += 1 in loop, so your results would be 0 at the beginning, then print results, not their len

Upvotes: 3

Related Questions