CODEWITHSUNDEEP
pythongeneratortqdm
George
George

Reputation: 2093

tqdm show progress for a generator I know the length of

I'm looping over a large file that I know the length of, but am processing lazily since it's too large to fit in memory. I'd like to be able to use tqdm to keep track of my progress through the file, but since it can't get the total number of examples out of the generator I'm using, the only thing it shows is the estimated iterations/second. Is there any way to tell tqdm how many elements it's going to be looping over total so I can get some of the other statistics?

Upvotes: 118

Views: 85595

Answers (2)

Seppo Enarvi
Seppo Enarvi

Reputation: 3663

If you provide an iterator that implements __len__(), tqdm will call that function to obtain the total number of iterations.

import time
from tqdm import tqdm

class File:
    def __init__(self):
        self.line = 0
    
    def __len__(self):
        return 5
    
    def __iter__(self):
        return self
    
    def __next__(self):
        self.line += 1
        if self.line > len(self):
            raise StopIteration()
        return self.line

for _ in tqdm(File()):
    time.sleep(1)

Otherwise, you can provide the total number of iterations in the total parameter.

class File:
    def __init__(self):
        self.line = 0
    
    def __iter__(self):
        return self
    
    def __next__(self):
        self.line += 1
        if self.line > 5:
            raise StopIteration()
        return self.line

for _ in tqdm(File(), total=5):
    time.sleep(1)

Upvotes: 4

O. Edholm
O. Edholm

Reputation: 2412

You can pass the length to the argument total to make it work.

Example:

from tqdm import tqdm

length = 1000000
generator = (3 * n for n in range(length))  # just doing something random
for n in tqdm(generator, total=length):
    pass

Upvotes: 193

Related Questions