CODEWITHSUNDEEP
pythongenerator
iTayb
iTayb

Reputation: 12743

How to efficiently iterate over multiple generators?

I've got three different generators, which yields data from the web. Therefore, each iteration may take a while until it's done.

I want to mix the calls to the generators, and thought about roundrobin (Found here). The problem is that every call is blocked until it's done.

Is there a way to loop through all the generators at the same time, without blocking?

Upvotes: 3

Views: 1069

Answers (2)

Ben Burns
Ben Burns

Reputation: 15206

In short, no: there's no good way to do this without threads.

Sometimes ORMs are augmented with some kind of peek function or callback that will signal when data is available. Otherwise, you'll need to spawn threads in order to do this. If threads are not an option, you might try switching out your database library for an asynchronous one.

Upvotes: 1

lunixbochs
lunixbochs

Reputation: 22395

You can do this with the iter() method on my ThreadPool class.

pool.iter() yields threaded function return values until all of the decorated+called functions finish executing. Decorate all of your async functions, call them, then loop through pool.iter() to catch the values as they happen.

Example:

import time
from threadpool import ThreadPool
pool = ThreadPool(max_threads=25, catch_returns=True)

# decorate any functions you need to aggregate
# if you're pulling a function from an outside source
# you can still say 'func = pool(func)' or 'pool(func)()
@pool
def data(ID, start):
    for i in xrange(start, start+4):
        yield ID, i
        time.sleep(1)

# each of these calls will spawn a thread and return immediately
# make sure you do either pool.finish() or pool.iter()
# otherwise your program will exit before the threads finish
data("generator 1", 5)
data("generator 2", 10)
data("generator 3", 64)

for value in pool.iter():
    # this will print the generators' return values as they yield
    print value

Upvotes: 5

Related Questions