CODEWITHSUNDEEP
python
Kevin Thompson
Kevin Thompson

Reputation: 2506

pythonic way to access each file in a list of directories

I have working code that looks like this:

# Wow. Much nesting. So spacebar
if __name__ == '__main__:
  for eachDir in list_of_unrelated_directories:
      for eachFile in os.listdir(eachDir):
          if eachFile.endswith('.json'):
            # do stuff here

I'd like to know if there is a more elegant way of doing that. I would like to not have my code nested three layers deep like that, and if I could get this to a one-liner like

for each file that ends with .json in all these directories:
  # do stuff

That would be even more awesome. I also edited this to point out that the directories are not all in the same folder. Like you might be looking for .json files in your home folder and also your /tmp folder. So I'm not trying to move recursively through a single folder.

Upvotes: 5

Views: 215

Answers (5)

Roberto
Roberto

Reputation: 9080

Surely, the following code is not Pythonic because is not the simplest or the most clear one, definitely doesn't follow the Zen of Python

However, It's a one-line approach and It was fun to do it ;-):

def manage_file(filepath):
    print('File to manage:', filepath)

EDIT: Based in accepted answer and I've updated my answer to use glob(), the result is still a bit Freak code but It's less code that my previous approach

map(manage_file, [fn for fn in sum((glob('%s/*.json' % eachDir) for eachDir in data_path), [])])

Upvotes: 1

Cody Piersall
Cody Piersall

Reputation: 8547

The most Pythonic way is (in my opinion) to write a function that yields the files of a certain type and use it. Then your calling code is very clear and concise. Some of the other answers are very concise but incredibly confusing; in your actual code you should value clarity over brevity (although when you can get both that is, of course, preferred).

Here's the function:

import os

def files_from_directories(directories, filetype):
    """Yield files of filetype from all directories given."""
    for directory in directories:
        for file in glob.glob(os.path.join(directory, '*' + filetype))
            yield file

Now your calling code really is a one-liner:

# What a good one-liner!!!!
for json_file in files_from_directories(directories, '.json'):
    # do stuff

So now you have a one-liner, and a very clear one. Plus if you want to process any other type of file, you can just reuse the function with a different filetype.

Upvotes: 1

Paulo Bu
Paulo Bu

Reputation: 29794

You can use a generator expression to remove the nested loops:

for json_file in (f for dir in list_of_unrelated_dirs 
                    for f in os.listdir(dir) 
                    if f.endswith('.json')):
    print json_file.

If you want to apply a function to them, you could even improve it removing the remaining for loop with map() function:

map(fun,
    (f for dir in list_of_unrelated_dirs 
       for f in os.listdir(dir) 
       if f.endswith('.json'))
)

Hope this helps!

Upvotes: 1

KobeJohn
KobeJohn

Reputation: 7545

It's not really less nesting, it's just nesting within a comprehension.

This gets all things that end with '.json' and are also confirmed as files (ignores folders that end with '.json').

Standalone Code

import os

unrelated_paths = ['c:/', 't:/']
json_files = (os.path.join(p, o) for p in unrelated_paths
              for o in os.listdir(p)
              if (o.lower().endswith('.json')
                  and os.path.isfile(os.path.join(p, o))))
for json_file in json_files:
    print json_file

Upvotes: 0

bavaza
bavaza

Reputation: 11037

glob() can reduce you to two levels:

for d in list_of_unrelated_directories:
    for f in glob(join(d, '*.json')):
         _process_json_file(f)

If your list_of_unrelated_directories is really a list of totally unrelated directories, I don't see how you can avoid the first loop.
If they do have something in common (say a common root, and some common prefix), then you can use os.walk() to traverse the tree and grab all matching files.

Upvotes: 0

Related Questions