CODEWITHSUNDEEP
pythonfunctionfunctional-programmingiteratorgenerator
Stiivi
Stiivi

Reputation: 2097

How to compose iterators?

I have a network of nodes passing structured data in between. For my subproblem, we have this branch - linear sequence of nodes:

nodes = [source, n1, n2, n3, n4]

First node is a generator, each other node gets values from input node and provides output value. Current implementation is plain get() from a pipe and put() to a pipe and there is separate thread per node (there is reason for that). I want to change it to a yield-ing iterator.

And I want to evaluate in a following way (if we consider node to be a callable):

for result in n4(n3(n2(n1(source()))):
    print result

I imagine the construction of evaluation context like this:

context = src
for node in nodes[1:]:
    context = pipe(context, node)

for result in context:
    print result

Limitations:

I still want to be able to use nodes separately - not nested, piping data by other means, because the nodes might be in separate threads. Example: [source, n1,n2] in one thread (might be nested), [n3, n4] in the other (might be nested), data piped between n2 and n3. Case: there might be a non-linear node graph where I want to group branches this way.

node has to be a class to hold computation state

How the implementation of the context and the pipe(context, node) might look like? Or if it can be solved in a different way, do you have any hints?

Can yield from in Python 3.3 (PEP380) help my case in any way?

Upvotes: 4

Views: 1510

Answers (1)

Marcin
Marcin

Reputation: 49856

If all you want is to compose arbitrary numbers of functions (or callables), use the compose_mult recipe from the functional module documentation.

A solution which uses that:

from functional import compose, foldr, partial
from itertools  import imap
compose_mult = partial(reduce, compose) 
chain_nodes = lambda nodes: imap(compose_mult(nodes[1:]), nodes[0])
chain_gen_nodes = lambda nodes: imap(compose_mult((g.send for g in nodes[1:])), nodes[0])


# equivalent not as a one-liner
#def chain_nodes(nodes):
#    source = nodes[0]
#    composed_nodes = compose_mult(nodes[1:])
#    return (composed_nodes(x) for x in source)

If the nodes are generators that accept input (via send), then use chain_gen_nodes, which extracts their send function.

Note, however, that one is not allowed to send to a just-started generator (because it has to be at the point of a yield to receive the value). This is something you are going to have to handle yourself, such as by having your generators yield a dummy value on their first iteration, and advancing them at some point before sending them to chain_nodes. Or you could just keep your nodes as ordinary callables.

If you do need to advance the iterators one step: next(izip(*nodes[1:]))

Upvotes: 2

Related Questions