Standard way to convert iterator-returning function into "proper iterable"-returning function?

Question

To avoid confusion, let me define:

proper iterable: an iterable object that is not an iterator.

---

Q: Does Python's Standard Library already provide a way to convert an "iterator-returning function" into a "proper iterable-returning function"?

---

I thought I'd seen this somewhere, but now I can't find it. In particular, I scanned through the docs for itertools, but did not spot it.

---

FWIW, this homegrown implementation seems to work:

def to_iterable_maker(iterator_maker):

    def iterable_maker(*args, **kwargs):
        class nonce_iterable(object):
            def __iter__(self):
                return iterator_maker(*args, **kwargs)
        return nonce_iterable()

    return iterable_maker

...but the one-time nonce_iterable class in there looks clumsy to me. I'm sure an implementation of such a thing from the Standard Library would be a lot better.

---

@Nikita

Try this:

import itertools

base = range(3)
poops_out = itertools.permutations(base)

print list(poops_out)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

print list(poops_out)
# []

myperms = to_iterable_maker(itertools.permutations)
keeps_going = myperms(base)

print list(keeps_going)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

print list(keeps_going)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

print list(keeps_going)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

There is a difference between the values returned by itertools.permutations and to_iterable_maker(itertools.permutations). My question is: does the Standard Library already provide something analogous to to_iterable_maker?

Answer 1

It doesn't make sense - "iterator-returning function" into an "iterable returning function". If a function is returning an iterator, then it's already returning an iterable, because iterators are iterables as they are required to have __iter__ method.

From the docs:

iterable

An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as list, str, and tuple) and some non-sequence types like dict, file objects, and objects of any classes you define with an __iter__() or __getitem__() method. Iterables can be used in a for loop and in many other places where a sequence is needed (zip(), map(), ...). When an iterable object is passed as an argument to the built-in function iter(), it returns an iterator for the object. This iterator is good for one pass over the set of values. When using iterables, it is usually not necessary to call iter() or deal with iterator objects yourself. The for statement does that automatically for you, creating a temporary unnamed variable to hold the iterator for the duration of the loop. See also iterator, sequence, and generator.

iterator

An object representing a stream of data. Repeated calls to the iterator’s __next__() method (or passing it to the built-in function next()) return successive items in the stream. When no more data are available a StopIteration exception is raised instead. At this point, the iterator object is exhausted and any further calls to its __next__() method just raise StopIteration again.

Iterators are required to have an __iter__() method that returns the iterator object itself so every iterator is also iterable

and may be used in most places where other iterables are accepted. One notable exception is code which attempts multiple iteration passes. A container object (such as a list) produces a fresh new iterator each time you pass it to the iter() function or use it in a for loop. Attempting this with an iterator will just return the same exhausted iterator object used in the previous iteration pass, making it appear like an empty container.

UPD:

What I mean is...

(Showing by steps to compare)

Case 1:

1. f = to_iterable_maker(iterator_maker);
2. i = f(some_var), i is nonce_iterable with __iter__;
3. j = iter(i), j is iterator returned by iterator_maker(some_var);
4. next(j), returns some value dependent on some_var.

Case 2:

1. f = iterator_maker;
2. i = f(some_var), i is iterator equal to iterator_maker(some_var), which has __iter__ (per iterator protocol);
3. j = iter(i), j is iterator returned by iterator_maker(some_var), because calling __iter__ on iterator returns itself, so j is i returns true;
4. next(j), returns some value dependent on some_var.

As you can see, nothing really changes, except for additional complications on the preparation step.

May be you could provide additional information on what you are trying to achieve by such 'wrapping', to understand the real issue.

As per your question, I can't think of any library function, that would turn iterator into iterable, because it already is. If you are trying to duplicate iterators may be take a look at itertools.tee().

UPD2:

So, now I see, that the goal is to convert single-pass iterator to multi-pass iterator...

My answer to:

"does the Standard Library already provide something analogous to to_iterable_maker?"

is "No". But the closest is itertools.tee(), which can help you clone a single iterator into several, that you can use after. Regarding your example:

import itertools

base = range(3)
poops_out = itertools.permutations(base)

iterators = itertools.tee(poops_out, 4)

#You shouldn't use original iterator after clonning, so make it refer to a clone
#to be used again, otherwise ignore the following line
poops_out, iterators = iterators[0], iterators[1:]

for it in iterators:
    print list(it)

#Prints:
#[(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]
#[(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]
#[(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

Another common way to get iterable from iterator is to convert it using list() or tuple(), which will allow multi-pass:

import itertools

base = range(3)

poops_out = itertools.permutations(base)

#Obviously poops_out gets consumed at the next line, so it won't iterate anymore
keeps_going = tuple(poops_out)

print list(poops_out)
# []

print list(poops_out)
# []

print list(keeps_going)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

print list(keeps_going)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

print list(keeps_going)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

Both methods described above might be heavy on memory usage, so sometimes it's not an option. In that case a solution you came to will work well. Another implemetation, which I could think of, and which is a little bit more object-oriented, but otherwise not much different from yours:

class IterableMaker(object):
    '''Wraps function returning iterator into "proper" iterable'''

    def __init__(self, iterator_maker):
        self.f = iterator_maker

    def __call__(self, *args, **kwargs):
        self.args = args
        self.kwargs = kwargs
        return self

    def __iter__(self):
        return self.f(*self.args, **self.kwargs)

the use is the same:

import itertools

class IterableMaker(object):

    def __init__(self, iterator_maker):
        self.f = iterator_maker

    def __call__(self, *args, **kwargs):
        self.args = args
        self.kwargs = kwargs
        return self

    def __iter__(self):
        return self.f(*self.args, **self.kwargs)

base = range(3)
poops_out = itertools.permutations(base)

print list(poops_out)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

print list(poops_out)
# []

my_perm = IterableMaker(itertools.permutations)

keeps_going = my_perm(base)

print list(keeps_going)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

print list(keeps_going)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

print list(keeps_going)
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]