CODEWITHSUNDEEP
pbsqsubtorque
David Parks
David Parks

Reputation: 32081

Understanding the -t option in qsub

The documentation is a bit unclear on exactly what the -t option is doing on a job submission using qsub

http://docs.adaptivecomputing.com/torque/4-0-2/Content/topics/commands/qsub.htm

From the documentation:

-t Specifies the task ids of a job array. Single task arrays are allowed. The array_request argument is an integer id or a range of integers. Multiple ids or id ranges can be combined in a comma delimited list. Examples: -t 1-100 or -t 1,10,50-100

Here's an example where things go wrong, I've requested 2 nodes, 8 processes per node, and an array of 16 jobs. Which I had hoped would be distributed naturally across the 2 nodes, but the 16 tasks were distributed ad-hoc across more than 2 nodes.

$ echo 'hostname' | qsub -q gpu -l nodes=2:ppn=8 -t 1-16
52727[]
$ cat STDIN.o52727-* | sort
gpu-3.local
gpu-3.local
gpu-3.local
gpu-3.local
gpu-5.local
gpu-5.local
gpu-5.local
gpu-5.local
gpu-5.local
gpu-5.local
gpu-7.local
gpu-7.local
gpu-7.local
gpu-7.local
gpu-7.local
gpu-7.local

Upvotes: 3

Views: 1068

Answers (1)

clusterdude
clusterdude

Reputation: 623

I suspect this will not completely answer your question, but what exactly you hope to accomplish remains unclear.

Specifying an array with qsub -t simply creates individual jobs, all with the same primary ID. Submitting the way you indicated will create 16 jobs, each requesting 16 total cores. This syntax merely makes it easier to submit a large number of jobs at once, without having to script a submission loop.

With Torque alone (i.e., disregarding the scheduler), you can force jobs to specific nodes by saying something like this:

qsub -l nodes=gpu-node01:ppn=8+gpu-node02:ppn=8

A more advanced scheduler can give you greater flexibility (e.g., Moab or Maui allow "-l nodes=2:ppn=8,nallocpolicy=exactnode", which applies NODEALLOCATIONPOLICY EXACTNODE to the job when scheduling, and will give you 8 cores each on exactly two nodes (ANY two nodes, in this case)).

Upvotes: 2

Related Questions