CODEWITHSUNDEEP
python
silverhawke
silverhawke

Reputation: 1

Processing several generators in parallel

Suppose I have several generators (which should be able to run in parallel). Is it possible to use multiprocessing module to call next() on these generators so that the processing would run in parallel?

I want to avoid making a list from the generators since it's very likely to consume lots of memory.

Context: Originally I have a generator which outputs all spanning trees of a given graph. Part of the algorithm involves iterating through the power set of a subset of the neighbors of a given vertex. I would like to parallelize this part, at least for the initial call. For a certain graph, it takes around half a second to output a tree for the first 1024 trees.

Upvotes: 0

Views: 311

Answers (1)

csinchok
csinchok

Reputation: 751

I think your main issue would be getting the data base to the parent process to build your graph. However, this could probably be accomplished by using a multiprocessing Queue.

A simple example:

import multiprocessing
from queue import Empty

def call_generator(generator, queue):
    for item in generator:
        queue.put(item)

def process_responses(queue):
    items = []
    while True:
        try:
            # After a one second timeout, we'll assume the generators are done
            item = queue.get(timeout=1)
        except Empty:
            print('done')
            break

        print('item: {}'.format(item))

generators = [
    iter(range(10)),
    iter(range(11, 20)),
    iter(range(20, 50))
]

queue = multiprocessing.Queue()

p = multiprocessing.Process(target=process_responses, args=(queue,))
p.start()


for generator in generators:
    generator_process = multiprocessing.Process(
        target=call_generator,
        args=(generator, queue)
    )
    generator_process.start()

p.join() # Wait for process_response to return

Upvotes: 1

Related Questions