CODEWITHSUNDEEP
pythonshellslurm
Tom Kurushingal
Tom Kurushingal

Reputation: 6496

Is there any way to run a secondary python script at regular intervals to work on output of a primary script in Slurm?

I am submitting a batch script that involves a primary command/script (an mpi process) that outputs data and I need to evaluate the progress of the primary process by running a secondary Python script at fixed intervals of time when the primary process is still running. Is there any command that would allow me to do so this with a Slurm batch script?

As an example, consider the primary process takes 24 hours, if I place the Python script normally after the end of the primary command/script, it would only run at the end of the primary process. I need the Python command/script to run every 1 hour to process data generated by the primary process. Is this possible on Slurm?

Upvotes: 0

Views: 72

Answers (1)

damienfrancois
damienfrancois

Reputation: 59260

The structure of the script would look like this:

#! /bin/bash
#SBATCH ...
#SBATCH ...

while : ; do sleep 3600 ; python <secondary script> ; done &

mpirun <primary command>

The idea is to run the secondary script in an infinite loop in the background. When the primary command finishes, the job is terminated and the background loop is stopped.

Upvotes: 1

Related Questions