CODEWITHSUNDEEP
bashjobsexecuteslurm
chippycentra
chippycentra

Reputation: 3432

execute bash files when previous ones have finished to run

Hello I would need help

In fact I need to execute several bash files ex:

file1.sh
file2.sh
file3.sh 
file4.sh

those file will generate data that will be used for another bash file call final.sh

So in order to gain time I want to execute the fileNb.sh files sumultany on a cluster by doing :

for file in file*.sh; do sbatch $file; done

, and then when all job have been done, I would like to execute automatically the final.sh file.

Does someone have an idea ?

Thank you very much

Upvotes: 1

Views: 744

Answers (3)

Mark Setchell
Mark Setchell

Reputation: 208052

You could do:

sbatch --wait file1.sh &
sbatch --wait file2.sh &
sbatch --wait file3.sh &
sbatch --wait file4.sh &
wait
sbatch final.sh

Or, more simply with GNU Parallel:

parallel -j4 sbatch --wait ::: file*.sh
sbatch final.sh

Upvotes: 2

damienfrancois
damienfrancois

Reputation: 59340

One clean option is to reorganise the set of jobs as a job array and then add a dependency for final job on the whole array.

Assuming fileN.sh looks like this:

#!/bin/bash
#SBATCH --<some option>
#SBATCH --<some other option>

./my_program input_fileN

you can make this a job array. In a single submission file file.sh, write this

#!/bin/bash
#SBATCH --<some option>
#SBATCH --<some other option>
#SBATCH --array=1-4

./my_program input_file${SLURM_ARRAY_TASK_ID}

Then run

JOBID=$(sbatch --parsable file.sh)
sbatch --dependency after:$JOBID final.sh

In case your jobs cannot be parametrised by an integer directly, create a Bash array like this:

#!/bin/bash
#SBATCH --<some option>
#SBATCH --<some other option>
#SBATCH --array=0-2

ARGS=(SRR63563 SRR63564 SRR63565)

fasterq-dump --threads 10  ${ARGS[$SLURM_ARRAY_TASK_ID]} -O /path1/path2/path3/

Upvotes: 5

Yuji
Yuji

Reputation: 525

Is this no good?

for file in file*.sh; do sbatch $file; done; ./final.sh

Upvotes: 0

Related Questions