CODEWITHSUNDEEP
pythonbash
Domovoi
Domovoi

Reputation: 11

Writing a bash file to run a python program for all folders in a directory

I currently wrote a program to take the standard deviation of a single set of data. I have over 200 folders, each with their own set of data. I am attempting to write a bash file that will execute this program for all folders (while outputting all of the standard deviation into a master file as dictated in python).

So far I have:

#!/bin/bash

for D in SAND; do python sample.py
  [ -d "$D" -a -x "$D/all" ] && "$D/all"
done

Note: SAND is my directory.

But this does not work. Please help.

In addition, when I try other examples and run them I keep having the error:

Traceback (most recent call last):
  File "sample.py", line 1, in <module>
    f=open("default")
IOError: [Errno 2] No such file or directory: 'default'

even though I DO have the data file of "default" in the folders.

Upvotes: 0

Views: 939

Answers (2)

Robᵩ
Robᵩ

Reputation: 168616

Alternatively, you could skip bash altogether and modify your Python script. os.walk() allows you to visit each directory in turn:

import os,sys

for arg in sys.argv[1:] or ['.']:
  for dirpath, _, filenames in os.walk(arg):
    for filename in filenames:
      if filename == 'all':
        all_file = os.path.join(dirpath, filename)
        default_file = os.path.join(dirpath, 'default')
        # ... whatever you do with SAND/foo/all
        # foo = open(all_file)
        # std_dev = bar(foo)
        # ... I'll just print them
        print all_file, default_file

Upvotes: 1

Charles Duffy
Charles Duffy

Reputation: 295315

The below assumes that SAND is the literal name of your directory.

First choice: Use a loop.

for d in SAND/*/all; do
  python sample.py "$d"
done

...or, if you need to change into the directory that's found...

orig_dir=$PWD
for d in SAND/*/all; do
  (cd "$d/.." && exec python "$orig_dir/sample.py" all)
done

Second choice: Use find.

I'd suggest searching directly for the targets named all:

find SAND -name all -exec python sample.py '{}' '+'

Alternately, with POSIX find, you can have find invoke a shell to perform more logic:

find SAND -type d -exec bash -c \
  'for d; do [[ -d "$d/all" ]] && python sample.py "$d/all"; done' _ '{}' +

If SAND is a variable name, not a literal directory, change SAND in the above to "$SAND", with the quotes (and, ideally, make it lower-case -- by convention, only environment variables and shell builtin variables should be all-caps to avoid namespace conflicts).

Upvotes: 2

Related Questions