CODEWITHSUNDEEP
pythonperformancegenerator
Dan
Dan

Reputation: 35473

Performance Considerations Using Multiple Layers of Generators in Python?

Are there any performance considerations for using a lot of generators chained together, as opposed to just a single generator.

For example:

def A(self, items):
    for item in self.AB(items):
        if object.A():
            yield item

def AB(self, items):
    for object in self.ABC(objects):
        if object.A() or object.B():
            yield object

def ABC(self, objects):
    for object in objects:
        if object.A() or object.B() or object.C():
            yield object

Clearly calling A(objects) is going to go through three different generators, but in many situations it makes the code re-use better if there are different generators to handle different filtering. Can anyone indicate that there is a significant impact on performance using this technique?

Upvotes: 3

Views: 297

Answers (1)

Andrew Wilkinson
Andrew Wilkinson

Reputation: 10846

There is nothing wrong with chaining generators, but in this example there is no reason for A to call self.AB, it can just loop over items to get the same result.

You should write your code as clearly as you can and if it's slow then use a profiler to determine where the bottleneck is. Contrived examples such as this one are too far from reality to be useful indicators of performance.

Upvotes: 2

Related Questions