How do I search through a list of files using a for loop in python

Question

See my code below. I keep getting this error when I run my code below:

"IndexError: list index out of range"

Code:

for x in range(0, numFiles):
    print(fileList[x])

for x in range(0, numFiles):
    f = open(dirName + "/" + fileList[x], 'r')  # open the file for reading
    fileText = f.read()                         # read file contents into string
    f.close()                                   # close file
    if fileText.find(tagName) == -1:            # if the file text doesn't contain the tag
        fileList.remove(fileList[x])            # then remove the file from the file list

The first for loop is here for debugging and it works as expected, but the second for loop where I am trying to actually open the file gives the index out of range error. Any help would be appreciated.

Answer 1

Thanks BorrajaX and other similar suggestions above, I decided to try a solution for a second time, but with a slightly different approach this time. Instead of removing from a copied list, I created a new empty list and appended to it if the tag name was found. And that worked great! Appreciate everyone's help here! Here's the modified code if anyone is interested.

for x in range(0, numFiles):
    print(fileList[x])

resultFileList = []

for x in range(0, numFiles):
    f = open(dirName + "/" + fileList[x], 'r')  # open the file for reading
    fileText = f.read()                         # read file contents into string
    f.close()                                   # close file
    if fileText.find(tagName) >= 0:            # if the file text doesn't contain the tag
        resultFileList.append(fileList[x])      # then remove the file from the file list

Answer 2

When you do fileList.remove you are making the list smaller if fileText.find(tagName) == -1 (You're changing the length of the list you're iterating over within the for loop)

See this simplified example:

test_list = [1, 2, 3, 4, 5]
num_items = len(test_list)

for i in range(0, num_items):
    print("Dealing with i=%s" % i)
    data = test_list[i]
    if data == 2 or data == 3 or data == 4:
        print("Removing i=%s (data=%s)" % (i, data))
        test_list.remove(data)
    print("Now test_list=%s, with %s items" % (test_list, len(test_list)))

Which outputs:

Dealing with i=0
Now test_list=[1, 2, 3, 4, 5], with 5 items
Dealing with i=1
Removing i=1 (data=2)
Now test_list=[1, 3, 4, 5], with 4 items
Dealing with i=2
Removing i=2 (data=4)
Now test_list=[1, 3, 5], with 3 items
Dealing with i=3
Traceback (most recent call last):
  File "./stack_101.py", line 25, in <module>
    data = test_list[i]
IndexError: list index out of range

Since you only have to "visit" the files once, I suggest you change your loop to a while:

test_list = [1, 2, 3, 4, 5]
num_items = len(test_list)

i = 0
while i < len(test_list):
    data = test_list[i]
    print("Dealing with i=%s (data=%s)" % (i, data))
    if data == 2 or data == 3 or data == 4:
        print("Removing i=%s, data=%s. NOT advancing" % (i, data))
        test_list.remove(data)
    else:
        i += 1
        print("Advancing counter to i=%s because we didn't remove the entry" % i)
    print("Now test_list=%s, with %s items" % (test_list, len(test_list)))
print("After the loop, test_list=%s" % test_list)

That correctly outputs:

Dealing with i=0 (data=1)
Advancing counter to i=1 because we didn't remove the entry
Now test_list=[1, 2, 3, 4, 5], with 5 items
Dealing with i=1 (data=2)
Removing i=1, data=2. NOT advancing
Now test_list=[1, 3, 4, 5], with 4 items
Dealing with i=1 (data=3)
Removing i=1, data=3. NOT advancing
Now test_list=[1, 4, 5], with 3 items
Dealing with i=1 (data=4)
Removing i=1, data=4. NOT advancing
Now test_list=[1, 5], with 2 items
Dealing with i=1 (data=5)
Advancing counter to i=2 because we didn't remove the entry
Now test_list=[1, 5], with 2 items
After the loop, test_list=[1, 5]

However: Do you really need to alter the list in place? As you can see, that messes up the code and leads to complications. How about just creating a new list with the non removed files?

Something like:

test_list = [1, 2, 3, 4, 5]
num_items = len(test_list)
new_list = []
for i in range(0, num_items):
    data = test_list[i]
    print("Dealing with i=%s (data=%s)" % (i, data))
    if not(data == 2 or data == 3 or data == 4):
        print("Keeping i=%s (data=%s)" % (i, data))
        new_list.append(data)
print("After the loop, new_list=%s" % new_list)

Which leaves the "proper" values in new_list:

Dealing with i=0 (data=1)
Keeping i=0 (data=1)
Dealing with i=1 (data=2)
Dealing with i=2 (data=3)
Dealing with i=3 (data=4)
Dealing with i=4 (data=5)
Keeping i=4 (data=5)
After the loop, new_list=[1, 5]

Applied to your code I guess it'd be something like this (untested):

found_files = []
for x in range(0, numFiles):
    f = open(dirName + "/" + fileList[x], 'r')  # open the file for reading
    fileText = f.read()                         # read file contents into string
    f.close()                                   # close file
    if fileText.find(tagName) >= 0:             # if the file text contains the tag
        found_files.append(fileList[x])         # then add it to the new list

Answer 3

As others pointed out, you are removing elements from the list, this the index goes beyond the current size of the list, which is less than numFiles.

A way of solving it is:

for x in range(0, numFiles):
    print(fileList[x])
last_index = numFiles
x = 0
while x < last_index:
    f = open(dirName + "/" + fileList[x], 'r')  
    fileText = f.read()                         
    f.close()                                   
    if fileText.find(tagName) == -1:            
        fileList.pop(x)   #pop is better, less ambiguous in case there is duplicates     
        last_index -= 1   #Decrement the end of the loop
     else:
        x += 1  #go to the next index only if you didn't remove an item

Answer 4

To avoid the out of range error, try something like:

for f in fileList:
    < CODE HERE > # f will be the actual file name now

And if you want to keep the index, try something like:

for i, f in enumerate(fileList):
    < CODE HERE > # i will be a counter and f will be the actual file name

Edit -- Sorry didn't even notice that you were dynamically changing the list size. That is where the index error is!