CODEWITHSUNDEEP
memorypbs
armen
armen

Reputation: 443

watching memory in PBS

I'm running a job on a cluster (using PBS) that runs out of memory. I'm trying to print the memory status for each node separately while my other job is running. I created a shell script and included a call to that script from inside my job submission script. But when I'm submitting my job it gives me permission denied error on the line that calls the script. I don't understand why do I get that error.

Secondly, I was thinking that I can have a 'watch free' or 'watch ps aux' in my script file but now I'm thinking if that will cause my submitted job to get stuck in memory-watching script and never continue to get to the main line that calls my parallel program.

After all, how can I achieve logging my memory in PBS for the jobs I'm submitting. My code is a C++ program using MRMPI (MPI MapReduce) library.

Upvotes: 2

Views: 6163

Answers (2)

armen
armen

Reputation: 443

NOTE: if your ssh access to computing nodes of the cluster is closed, this method won't work!

This is how I ended up doing this. It might not be the best way but it works: In summary, I added some short sleep periods in between my map and reduce steps by calling c++ sleep() function. And also wrote a script that ssh's to the nodes my job is running on and then gets the memory status on those nodes writing them in a file (using 'free' or 'top' commands).

More detailed: in my PBS job script, somewhere before the call to my binary, I added this line:

#this goes in job script, before the call to the job binary:
cat $PBS_NODEFILE > /some/path/nodelist.log

This writes a list of the nodes that my job runs on, into a file.

I have a second script "watchmem.sh":

#!/bin/bash
for i in $(seq 60)
do
    while read line; 
    do
            ssh $line 'bash -s' < /some/path/remote.sh "$line"
    done < /some/path/nodelist.log
    sleep 10
done

This script reads the file nodelist.log that we generated before, performs an ssh into each node and calls a third (and last script), remote.sh, on each of those nodes.

remote.sh contains the commands that we run on every node of our job. In this case it prints the current time and the result of 'free' into separate files for each node:

#remote.sh
echo "Current time : $(date)" >> $1
free >> $1  #this can be replaced by top by specifying a -n for it

Comparing the times from these files and the times I'm printing from my binary let's me find out the memory consumption (alloc/dealloc) in each step. The sleep periods in my job is to make sure my scripts capture the memory status in between steps. 'sleep 10' in my script is to avoid unnecessary writes to the file; this period should be comparable to the sleep duration in the main job.

Upvotes: 0

dbeer
dbeer

Reputation: 7203

To see how much memory is being used throughout the job, run qstat -f:

$ qstat -f | grep used
    resources_used.cput = 00:02:51
    resources_used.energy_used = 0
    resources_used.mem = 6960kb
    resources_used.vmem = 56428kb
    resources_used.walltime = 00:01:26

To examine past jobs you can look in the accounting file. This is located in the server_priv/accounting directory, the default is /var/spool/torque/server_priv/accounting/.

The entries look like this:

09/14/2015 10:52:11;E;202.napali;user=dbeer group=company jobname=intense.sh queue=batch ctime=1442248534 qtime=1442248534 etime=1442248534 start=1442248536 owner=dbeer@napali exec_host=napali/0-2 Resource_List.neednodes=1:ppn=3 Resource_List.nodect=1 Resource_List.nodes=1:ppn=3 session=20415 total_execution_slots=3 unique_node_count=1 end=0 Exit_status=0 resources_used.cput=1989 resources_used.energy_used=0 resources_used.mem=9660kb resources_used.vmem=58500kb resources_used.walltime=995

Upvotes: 3

Related Questions