CODEWITHSUNDEEP
pythonlazy-evaluationgenerator
jsbueno
jsbueno

Reputation: 110156

What is the most pythonic way to have a generator expression executed?

More and more features of Python move to be "lazy executable", like generator expressions and other kind of iterators. Sometimes, however, I see myself wanting to roll a one liner "for" loop, just to perform some action.

What would be the most pythonic thing to get the loop actually executed?

For example:

a = open("numbers.txt", "w")
(a.write ("%d " % i) for i in xrange(100))
a.close()

Not actuall code, but you see what I mean. If I use a list generator, instead, I have the side effect of creating a N-lenght list filled with "None"'s.

Currently what I do is to use the expression as the argument in a call to "any" or to "all". But I would like to find a way that would not depend on the result of the expression performed in the loop - both "any" and "all" can stop depending on the expression evaluated.

To be clear, these are ways to do it that I already know about, and each one has its drawbacks: 

[a.write ("%d " % i) for i in xrange(100))]

any((a.write ("%d " % i) for i in xrange(100)))

for item in (a.write ("%d " % i) for i in xrange(100)): pass

Upvotes: 4

Views: 794

Answers (4)

jsbueno
jsbueno

Reputation: 110156

It is 2019 - and this is a question from 2010 that keeps showing up. A recent thread in one of Python's mailing lists spammed over 70 e-mails on this subject, and they refused again to add a consume call to the language.

On that thread, the most efficient mode to that actually showed up, and it is far from being obvious, so I am posting it as the answer here:

import deque

consume = deque(maxlen=0).extend

And then use the consume callable to process generator expressions.

It turns out the deque native code in cPython actually is optimized for the maxlen=0 case, and will just consume the iterable.
The any and all calls I mentioned in the question should be equally as efficient, but one has to worry about the expression truthiness in order for the iterable to be consumed.

I see this still may be controversial, after all, an explicit two line for loop can handle this - I remembered this question because I just made a commit where I create some threads, start then, and join then back - without a consume callable, that is 4 lines with mostly boiler plate, and without benefiting from cycling through the iterable in native code: https://github.com/jsbueno/extracontext/blob/a5d24be882f9aa18eb19effe3c2cf20c42135ed8/tests/test_thread.py#L27

Upvotes: 1

Jerub
Jerub

Reputation: 42608

There is one obvious way to do it, and that is the way you should do it. There is no excuse for doing it a clever way.

a = open("numbers.txt", "w")
for i in xrange(100):
    a.write("%d " % i)
d.close()

Lazy execution gives you a serious benefit: It allows you to pass a sequence to another piece of code without having to hold the entire thing in memory. It is for the creation of efficient sequences as data types.

In this case, you do not want lazy execution. You want execution. You can just ... execute. With a for loop.

Upvotes: 10

Alex Martelli
Alex Martelli

Reputation: 881497

There are many accumulators which have the effect of consuming the whole iterable they're given, such as min or max -- but even they don't ignore entirely the results yielded in the process (min and max, for example, will raise an exception if some of the results are complex numbers). I don't think there's a built-in accumulator that does exactly what you want -- you'll have to write (and add to your personal stash of tiny utility function) a tiny utility function such as

def consume(iterable):
    for item in iterable: pass

The main reason, I guess, is that Python has a for statement and you're supposed to use it when it fits like a glove (i.e., for the cases you'd want consume for;-).

BTW, a.write returns None, which is falsish, so any will actually consume it (and a.writelines will do even better!). But I realize you were just giving that as an example;-).

Upvotes: 6

Darius Bacon
Darius Bacon

Reputation: 15124

If I wanted to do this specific example, I'd write

for i in xrange(100): a.write('%d ' % i)

If I often needed to consume an iterator for its effect, I'd define

def for_effect(iterable):
    for _ in iterable:
        pass

Upvotes: 7

Related Questions