CODEWITHSUNDEEP
bash
xyz0o
xyz0o

Reputation: 13

A handy script to execute a program for all files in a directory one by one

I have 61 .fastq files, I want to run an app for each one of them separately, resulting in 61 output files, so how to write a script avoiding to list all 61 files? I tried a couple of simple scripts as I am not a pro, but when running, the program fails to open fastq files, however, when I run the program for only one file, there is no problem in opening it. do you think, technically speaking, the script below is correct and should work? or am I missing something?

samples="$(ls data)"
echo ${samples} | tr " " "\n" | while read sample; do
    gsnap -D ./cinxia  ./data/${sample}.fastq ; done

OR

for sample in /data/*.fastq; do
    gsnap -D ./cinxia  ./data/${sample}.fastq ; done

I expect the program gsnap runs for each one of my 61 fastq file and gives me a result each time with the same file name

Upvotes: 1

Views: 145

Answers (2)

slaiyer
slaiyer

Reputation: 464

While izzy's answer is correct and straightforward, if you're looking for a performance boost (pending correctness checks and benchmarking), you may wish to consider the following approach:

find data -name *.fastq -type f -maxdepth 1 -print0 2>/dev/null \
  | xargs -0 -n1 -I{} -P"$(nproc)" gsnap -D ./cinxia "{}"

xargs -P<num> can help parallelize gsnap invocation to <num> levels, where nproc tells it the number of processing units available to the current process (which may be less than the number of online processors).

Upvotes: 1

izzy
izzy

Reputation: 807

The second one is better, it's bad form to sniff the output of ls. I think if you have certain characters in the file names you could get in trouble, but as long as you know the input file names are "normal" you can get away with just using quotes, like so:

for sample in /data/*.fastq; do
    gsnap -D ./cinxia "./data/${sample}.fastq"
done

The quotes will make it so, for example, if your file name has spaces it runs:

gsnap -D ./cinxia "./data/file name with spaces.fastq"

You can see that without the quotes, the command line would see that is many words and not one file. Be careful when dealing with quoting and expansion in bash, it's confusing! In some situations, you get bad results even when using quotes.

What would happen if you had a file named badfile; rm -rf /?

Upvotes: 2

Related Questions