CODEWITHSUNDEEP
pythonlinuxbashunix
clickwerse
clickwerse

Reputation: 45

Running python scripts for different input directory through bash terminal

I am trying to automate my task through the terminal using bash. I have a python script that takes two parameters (paths of input and output) and then the script runs and saves the output in a text file.

All the input directories have a pattern that starts from "g-" whereas the output directory remains static. So, I want to write a script that could run on its own so that I don't have to manually run it on hundreds of directories.

$ python3 program.py ../g-changing-directory/    ~/static-directory/ > ~/static-directory/final/results.txt

Upvotes: 0

Views: 263

Answers (2)

lab9
lab9

Reputation: 586

There are many ways to do this, here's what I would write:

find .. -type d -name "g-*" -exec python3 program.py {} ~/static-directory/ \; > ~/static-directory/final/results.txt

You haven't mentioned if you want nested directories to be included, if the answer is no then you have to add the -maxdepth parameter as in @toydarian's answer.

Upvotes: 0

toydarian
toydarian

Reputation: 4584

You can do it like this:

find .. -maxdepth 1 -type d -name "g-*" | xargs -n1 -P1 -I{} python3 program.py {} ~/static-directory/ >> ~/static-directory/final/results.txt

find .. will look in the parent directory -maxdepth 1 will look only on the top level and not take any subdirectories -type d only takes directories -name "g-*" takes objects starting with g- (use -iname "g-*" if you want objects starting with g- or G-).
We pipe it to xargs which will apply the input from stdin to the command specified. -n1 tells it to start a process per input word, -P1 tells it to only run one process at a time, -I{} tells it to replace {} with the input in the command.
Then we specify the command to run for the input, where {} is replaced by xargs.: python3 program.py {} ~/static-directory/ >> ~/static-directory/final/results.txt have a look at the >> this will append to a file if it exists, while > will overwrite the file, if it exists.

With -P4 you could start four processes in parallel. But you do not want to do that, as you are writing into one file and multi-processing can mess up your output file. If every process would write into its own file, you could do multi-processing safely.

Refer to man find and man xargs for further details.

There are many other ways to do this, as well. E.g. for loops like this:

for F in $(ls .. | grep -oP "g-.*"); do 
  python3 program.py $F ~/static-directory/ >> ~/static-directory/final/results.txt
done

Upvotes: 1

Related Questions