CODEWITHSUNDEEP
pythongeneratorpython-asyncio
Rotareti
Rotareti

Reputation: 53973

Python asyncio - How to chain() the results returned from as_completed() lazily?

In the following code example I need to find a solution for Step 3):

import asyncio as aio
import random


async def produce_numbers(x):
    await aio.sleep(random.uniform(0, 3.0))
    return [x * x, x * x + 1, x * x + 2]


async def main():

    # Step 1) Create a list of unawaited coroutines.
    coros = [produce_numbers(i) for i in range(0, 10)]

    # Step 2) Create generator to return items in order of completion.
    futs = aio.as_completed(coros)  # type: generator

    # Step 3) Create a generator that returns every single number lazily.
    # This is the step I don't know how to accomplish.
    # I can't find a way to chain the lists that I get from each
    # `produce_numbers()` call together in a lazy manner.

    # Step 4) This loops should fire as soon as the first
    # `produce_numbers()` coroutine finished.
    for number in my_number_generator:
        print(number)


if __name__ == '__main__':
    loop = aio.get_event_loop()
    loop.run_until_complete(main())

Perhaps there is a crucial function in the standard lib that I'm missing?

Upvotes: 0

Views: 2675

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1125308

asyncio.as_completed() yields futures, and you’d need to await each to get your list result. You could use an async generator expression with a double loop to flatten them:

flattened = (v for fut in futs for v in await fut)

async for number in flattened:
    print(number)

Or use a separate async generator function to do the looping with full statements:

async def flatten_futures(futures):
    for future in futures:
        for value in await future:
            yield value

and in main() use this as

async for number in flatten_futures(futs):
    print(number)

Upvotes: 2

Related Questions