CODEWITHSUNDEEP
serverresourcesslurm
spiridon_the_sun_rotator
spiridon_the_sun_rotator

Reputation: 1044

Is there a way to figure out how many gpu's on a node via slurm?

I am working with a SLURM workload manager, and we have nodes with 4 GPUs.

The are several possible states of a node:

Canonical way to determine the resource utilization is sinfo command - https://slurm.schedmd.com/sinfo.html. There are several flags and options. However, it seems, that none of them can tell how many resources on a given node are allocated ath the moment.

Say, I would like to know, for the mixed state, whether 1,2 or 3 GPU's are available. Or this information is confidential and for some reason not available to users?

Upvotes: 9

Views: 15610

Answers (1)

damienfrancois
damienfrancois

Reputation: 59250

If you run

scontrol show nodes

as a regular user, you will see a lot of information about the nodes, among which the line that look like

 AllocTRES=cpu=8,mem=48G,gres/gpu=2

tells you how many GPUs are allocated: gres/gpu=2. The other line

 CfgTRES=cpu=64,mem=257707M,billing=64,gres/gpu=2

tells how many GPUs are configured: gres/gpu=2. With these two lines, you can deduce the amount of GPUs still available on the node.

Upvotes: 13

Related Questions