IndexError: list index out of range, but list length OK

Question

New to programming, looking for a deeper understanding on whats happening.

Goal: open a file and print the first 10 lines. (similar to head command)

Code:

with open('file') as f: 
      for i in range(0,10): 
          print([line.strip('\n') for line in f][i])

Result: prints first line fine, then returns the out of range error

File: Is a simple text file with 20 lines, no more than 50 chars per line

FYI - Removed range line and printed both type(list) and length(20). Printed specific indexes without issue (unless >1 in a row)

Able to get the desired result with different code, but trying to improve using with/as

Answer 1

You can actually iterate over a file. Which is what you should be doing here.

with open('file') as f:
    for i, line in enumerate(file, start=1):
        # Get out of the loop if we hit 10 lines
        if i >= 10:
            break
        # Line already has a '\n' at the end
        print(line, end='')

The reason that your code is failing is because of your list comprehension:

[line.strip('\n') for line in f]

The first time through your loop that consumes all of the lines in your file. Now your file has no more lines, so the next time through it creates a list of all the lines in your file and tries to get the [1]st element. But that doesn't exist because there are no lines at the end of your file.

If you wanted to keep your code mostly as-is you could do

lines = [line.rstrip('\n') for line in f]
for i in range(10):
    print(lines[i])

But that's also silly, because you could just do

lines = f.readlines()

But that's also silly if you just want up to the 10th line, because you could do this:

with open('file') as f:
    print('\n'.join(f.readlines()[:10]))

---

Some further explanation:

The shortest and worst way you could fix your code is by adding one line of code:

with open('file') as f: 
      for i in range(0,10):
          f.seek(0)  # Add this line
          print([line.strip('\n') for line in f][i])

Now your code will work - but this is a horrible way to get your code to work. The reason that your code isn't working the way you expect in the first place is that files are consumable iterators. That means that when you read from them eventually you run out of things to read. Here's a simple example:

import io

file = io.StringIO('''
This is is a file
It has some lines
okay, only three.
'''.strip())

for line in file:
    print(file.tell(), repr(line))

This outputs

18 'This is is a file\n'
36 'It has some lines\n'
53 'okay, only three.'

Now if you try to read from the file:

print(file.read())

You'll see that it doesn't output anything. That's because you've "consumed" the file. I mean obviously it's still on disk, but the iterator has reached the end of the file. But as shown, you can seek in the file.

print(file.tell())
file.seek(0)
print(file.tell())
print(file.read())

And you'll see your entire file printed. But what about those other positions?

file.seek(36)
print(file.read())  # => okay, only three.

As a side note, you can also specify how much to read:

file.seek(36)
print(file.read(4))  # => okay
print(file.tell())  # => 40

So when we read from a file or iterate over it we consume the iterator and get to the end of the file. Let's put your new tools to work and go back to your original code and explore what's happening.

with open('file') as f:
    print(f.tell())
    lines = [line.rstrip('\n') for line in f]
    print(f.tell())
    print(len([line for line in f]))
    print(lines)

You'll see that you're at a different location in the file. And the second list comprehension produces an empty list. That's because when a list comprehension is evaluated it executes immediately. So when you do this:

for i in range(10):
    print([line.strip('\n') for line in f][i])

What you're doing the first time, i = 0 and then the list comprehension reads to the end of the file. Now it takes the [0]th element of the list, or the first line in the file. But your file iterator is at the end of the file.

So now we get back to the beginning of the list and i = 1. Now we iterate to the end of the file, but we're already at the end so there are no lines to read, and we've got an empty list [] that we try to get the [0]th element of. But there's nothing there. So we get an IndexError.

List comprehensions can be useful, but when you're beginning it's usually much easier to write a for loop and then turn it into a list comprehension. So you might write something like this:

with open('file') as f:
    for i, line in enumerate(file, start=10):
        if i < 10:
            print(line.rstrip())

Now, we shouldn't print inside a list comprehension, so instead we'll collect everything. We start out by putting what we want:

[line.rstrip()

Now add the for bit:

[line.rstrip() for i, line in enumerate(f)

And finally add the filter and our closing brace:

[line.rstrip() for i, line in enumerate(f) if i < 10]

For more on list comprehensions, this is a fantastic resource: http://treyhunner.com/2015/12/python-list-comprehensions-now-in-color/