CODEWITHSUNDEEP
python
Tedd Johnson
Tedd Johnson

Reputation: 81

how to do a loop in chunks in python

I have a file with 100 lines. I am reading it into python

Input = [line.rstrip() for line in open("input.txt")]

Input has 100 lines, but I have a function that can only handle 20 at at time.

How do I do something like:

for (20 lines) in Input as lines: magic_function(lines)

I think the term I am looking for is either chunking or iterating? But I feel like I am not searching right because the info I m finding seems more complex than it should be.

Upvotes: 0

Views: 1758

Answers (2)

Scratch'N'Purr
Scratch'N'Purr

Reputation: 10409

If your file is really big then running Input = [line.rstrip() for line in open("input.txt")] will use a lot of memory. It's better to stream it while reading 20 lines at a time:

lines = []
for idx, l in enumerate(open("input.txt")):
    lines.append(l.rstrip())
    if (idx + 1) % 20 == 0:
        for line in lines:
            # process
        lines = []  # reset
# process any last remaining lines that wasn't caught in the modulo if
for line in lines:
    # process

Upvotes: 0

Rakesh
Rakesh

Reputation: 82765

Try slicing your main list to sub-lists of 20 and then process it.

Ex:

Input = [line.rstrip() for line in open("input.txt")]
InPut = [Input[line:line+20] for line in range(0, len(Input), 20)]
for chunk in Input:
    for line in chunk:
        #process

Upvotes: 2

Related Questions