CODEWITHSUNDEEP
pythonasynchronousgeneratoryield
miditower
miditower

Reputation: 107

How to make a Python generator execution asynchronous?

I have a piece of of code that looks like this:

def generator():
    while True:
        result = very_long_computation()
        yield result

def caller():
    g = generator()
    for i in range(n):
        element = next(g)
        another_very_long_computation()

Basically, I'd like to overlap the execution of very_long_computation() and another_very_long_computation() as much as possible.

Is there simple way to make the generator asynchronous? I'd like the generator to start computing the next iteration of the while loop right after result has been yielded, so that (ideally) the next result is ready to be yielded before the succesive next() call in caller().

Upvotes: 1

Views: 130

Answers (1)

user2357112
user2357112

Reputation: 280688

There is no simple way, especially since you've got very_long_computation and another_very_long_computation instead of very_slow_io. Even if you moved generator into its own thread, you'd be limited by CPython's global interpreter lock, preventing any performance benefit.

You could move the work into a worker process, but the multiprocessing module isn't the drop-in replacement for threading it likes to pretend to be. It's full of weird copy semantics, unintuitive restrictions, and platform-dependent behavior, as well as just having a lot of communication overhead.

If you've got I/O along with your computation, it's fairly simple to shove the generator's work into its own thread to at least get some work done during the I/O:

from queue import Queue
import threading

def worker(queue, n):
    gen = generator()
    for i in range(n):
        queue.put(next(gen))

def caller():
    queue = Queue()
    worker_thread = threading.Thread(worker, args=(queue, n))
    worker_thread.start()
    for i in range(n):
        element = queue.get()
        another_very_long_computation()

Upvotes: 3

Related Questions