CODEWITHSUNDEEP
bashcluster-computingqsub
solalito
solalito

Reputation: 1209

Send multiple qsub jobs with different runtime parameters

Last night, I sent a large number of jobs with qsub of the same executable but with different input parameters. Most of the jobs were in queue, waiting for the others to finish. This morning, I realized that all the jobs that were in queue used the last instance of my input file.

What is the standard way of working around this issue? Should I have one input file per job and compile my code so it reads the correct one? Or is there a better/more robust solution?

Upvotes: 1

Views: 826

Answers (1)

gauteh
gauteh

Reputation: 17205

You could create a master PBS script which loops over the different input paramters, executes them either in parallel or sequentially:

this simply gives executable a different input number for each job (IN), you should change this to loop over one or more of your input parameters as needed.

# PBS -l mppwidth=2048

NIN=10 # number of input parameters

for IN in `seq -w 1 $NIN`; do
   cd "sub_job_${IN}"
   executable $IN # runs jobs sequentially (you might have to prefix this with aprun)
done

or in parallel:

# PBS -l mppwidth=2048
# ^^ these should now be shared among the jobs.

NIN=10 # number of input parameters

for IN in `seq -w 1 $NIN`; do
   cd "sub_job_${IN}"
   executable $IN & # runs the job in the background, you might 
                    # have to prefix this with `aprun -n .. -N ..` or something
                    # so that each job only uses a portion of the total
                    # requested CPUs.
done
wait # wait for all jobs to finish

Upvotes: 1

Related Questions