CODEWITHSUNDEEP
slurm
meTchaikovsky
meTchaikovsky

Reputation: 7676

How can I know the real-time memory usage of a running job on slurm?

I know little about how cpu communicates with memories, so I’m not sure whether this is a ‘correct’ question to ask.

In a job script I submit to a slurm cluster, the script needs to read data from a database stored in the working dictionary. I want to monitor the memory used by running this script.

How can I write a bash script to do this? I have tried @CoffeeNerd's script. However, while the job is running, there is only one line of output in the file

AveCPU|AveRSS|MaxRSS

How can I modify this script to output the real-time memory usage?

I know sstat command, but I'm not sure whether something like sstat -j $JOBID.batch --format=MaxVMSize is the solution to my problem.

Upvotes: 8

Views: 4959

Answers (1)

damienfrancois
damienfrancois

Reputation: 59340

Slurm has a plugin that records a 'profile' of a job (PCU usage, memory usage, etc) into a HDF5 file. It holds a time series for each item measured.

Use

#SBATCH --profile=<all|none|[energy[,|task[,|filesystem[,|network]]]]>

to activate it.

See the documentation here.

Upvotes: 2

Related Questions