CODEWITHSUNDEEP
pythonmultithreadinghpc
Brian Andersen
Brian Andersen

Reputation: 71

Determine number of processors assigned to job

I am trying to right a python program that will use a pool of workers to perform tasks on a Linux hpc computing environment. I want to set the number of workers in the pool equal to the number of threads that I have requested for my job.

Commands such as multiprocessing.cup_count() and num_procs return 64, which is the number of threads on a node, but not the number of threads assigned to my job.

My current solution is to grep the command in the batch file for designating number of threads, and then examining the output:

os.system('grep -i "batch -n" * > output')

file_ = open('output','r')
file_lines = file_.readlines()
file_.close()

for line in file_lines:
  elems = line.strip().split()
  if '-n' in elems:
    position = elems.index('-n')
    num_procs = elems[position+1]

Is there a more succinct and straightforward way of doing this though?

Upvotes: 3

Views: 291

Answers (1)

Katia
Katia

Reputation: 3914

Depending on the scheduler you use, you can query the number of cores your job requested from the scheduler. For SGE - $NSLOTS; for SLURM - $SLURM_CPUS_PER_TASK; for PBS - $PBS_NP.

For example for SGE:

import os
#for SGE:
nslots = int(os.environ['NSLOTS']))

# for SLURM:
nslots = int(os.environ['SLURM_CPUS_PER_TASK']))

Upvotes: 1

Related Questions