CODEWITHSUNDEEP
pythonlistpython-os
TheTank
TheTank

Reputation: 505

Why does appending to a list take forever?

I wrote the following code:

import fnmatch
ll = []
for items in src:
    for file in os.listdir('/Users/swaghccc/Downloads/PulledImages/'):
        if fnmatch.fnmatch(file, items.split('/')[-1]):
            print file
            ll.append(file)

my src list contains paths to images. something like:

/path/to/image.jpg

These images are a subset of the images contained in the directory PulledImages.

The printing of the matched images works correctly. But when I try to put those imagesnames into a list ll it takes forever.

What on earth am I doing wrong?

Upvotes: 0

Views: 460

Answers (1)

Charles Duffy
Charles Duffy

Reputation: 295579

Appending doesn't take forever. Searching through a list, however, takes more time the longer your list is; and os.listdir(), being an operating system call, can be unavoidably slow when running against a large directory.

To avoid that, use a dictionary or set, not a list, to track the names you want to compare against -- and build that set only once, outside your loop.

# run os.listdir only once, storing results in a set for constant-time lookup
import sets
files = sets.Set(os.listdir('/Users/swaghccc/Downloads/PulledImages/'))

ll = []
for item in src:
    if item.split('/')[-1] in files:
        ll.append(file)

Community Wiki because I don't believe this question to be within topic guidelines without a MCVE; thus, not taking rep/credit for this answer.

Upvotes: 3

Related Questions