How to source another bash script if my script is being executing by SLURM?

Question

I have script for running my parallel program on cluster. I run it with usual command:

sbatch -p PARTITION -t TIME -N NODES /full/path/to/my/script.sh PARAMETERS-LIST

Inside that script.sh I need to source another bash script (which is located in the same directory where script.sh resides) to load some routines/variables. For my usual scripts which are executed on local computer I use the following:

SCRIPTDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd )"
source "$SCRIPTDIR/funcs.sh"
print_header "Some text"

and it works just fine. However, on cluster this doesn't work and I get the following error (just for example):

/var/tmp/slurmd/job1043319/slurm_script: line 9: /var/tmp/slurmd/jobID/funcs.sh: No such file or directory
/var/tmp/slurmd/job1043319/slurm_script: line 13: print_header: command not found

Seems like SLURM creates its own copy of script to be submitted and because of this I can't source any local scripts/files.

What can be done in that situation? It would be great if I can avoid hard-coding absolute paths inside my scripts...

Answer 1

The problem is that the location of the sbatch shell script, and only this script, is different in the case you just run it from your desktop's command prompt form the case of slurmstepd running it on a node. This happens because sbatch physically copies your script to every head node of the allocation, and runs it from there, using Slurm's fast hierarchical network topology mechanism. The end effect of this is that while the current directory is propagated to the script execution environment, the path to script differs (and can be different on different nodes). Let me explain using your example.

What is going on?

Of course, the script that you are including must be seen as the same file at the same location in the filesystem tree (on an NFS mount, normally). In this example, I assume that your username is bob (simply because it's most certainly not), and that your home directory /home/bob is mounted from an NFS export on every node, as well as your own machine.

Reading your code, I understand that the main script script.sh and the sourced file funcs.sh are located in the same directory. For simplicity, let's put them right into your home directory:

$ pwd
/home/bob
$ ls
script.sh funcs.sh

Let me also modify the script.sh as follows: I'm going to add the pwd line to see where we are, and remove the rest past the failing . builtin, as that is irrelevant anyway.

#!/bin/bash
pwd
SCRIPTDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd )"

The local run

Whichever directory is current is irrelevant, so let's complicate our test a bit by specifying a relative path to the script, even though it is in the current directory:

$ ../bob/script.sh PARAMETERS-LIST

In this case, the script is evaluated by bash as follows (step-by step, with the command stdout, variable expansion result or variable assigned value shown at each other line prefixed with a =>.

pwd
 => '/home/bob'

# Evaluate: SCRIPTDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd )"
${BASH_SOURCE[0]}
 => '../bob/script.sh'
dirname '../bob/script.sh'
 => '../bob'
cd '../bob'
 => Success, $? is 0
pwd
 => '/home/bob'
SCRIPTDIR='/home/bob'

# Evaluate: source "$SCRIPTDIR/funcs.sh"
$SCRIPTDIR
 => '/home/bob'
source '/home/bob/funcs.sh'
 => (Successfully sourced)

Here, your intended behavior of sourcing funcs.sh from the same directory where script.sh lives worked just fine.

The Slurm run

Slurm copies your script.sh to the spool directory on a node, and then executes it from there. If you specify the -D switch to sbatch, the current directory will be set to that (or to the value of $TMPDIR if that fails; or to /tmp is that, in turn, fails). if you do not specify the -D, the current directory is used. For now, suppose that /home/bob is mounted on the node, and that you simply submit your script without the -D:

$ sbatch -N1 ./script.sh PARAMETERS-LIST

Slurm allocates a node machine for you, copies the contents of your script ./script.sh into a local file (it happened to be named /var/tmp/slurmd/job1043319/slurm_script in your example), sets the current directory to /home/bob and executes the script file /var/tmp/slurmd/job1043319/slurm_script. I think you already understand what is going to happen.

pwd
 => '/home/bob'

# Evaluate: SCRIPTDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd )"
${BASH_SOURCE[0]}
 => '/var/tmp/slurmd/job1043319/slurm_script'
dirname '/var/tmp/slurmd/job1043319/slurm_script'
 => '/var/tmp/slurmd/job1043319'
cd '../bob'
 => Success, $? is 0
pwd
 => '/home/bob'
SCRIPTDIR='/var/tmp/slurmd/job1043319'

I think we should stop here. You already see that your assumed invariant of the main script and its sourced file being in the same directory is violated. Your script relies on this invariant, and therefore breaks.

So how do I solve this?

It depends on your requirements. You did not state any, but I can give a few suggestions that may align with your goals to a different degree each. This may have a positive side of my answer being useful to a wider SO audience.

OPTION 1. Enter into a binding agreement with yourself (and, if any, other users of your script) to always launch your script while in a particular directory.

In practice, this is the approach taken e. g. by a well-known speech recognition toolkit Kaldi¹: any script, any command you run, you must run from the experiment's root directory (link to example experiment).

If this approach is feasible, then anything you source, you source from the current directory (and/or a well-known path under it); example 1, top-level ./run.sh in the main experiment directory²

. ./cmd.sh
. ./path.sh

example 2, from a utility file utils/nnet/subset_data_tr_cv.sh in a directory that is itself soft-linked from the main experiment directory:

. utils/parse_options.sh

None of these . statements would work in any script invoked from an unconventional directory:

$ pwd
/home/bob/kaldi/egs/fisher_english/s5
$ utils/nnet/some_utility_script.sh  # This works.
$ cd utils/nnet
$ ./some_utility_script.sh           # This fails, by design.

Pros: Readable code. When you have 3,000 bash files totaling 600,000 lines of code, as our case at point does, this is important.
Pros: The code is very HPC-cluster-agnostic, and almost all scripts can run on your machine, with or without local multicore parallelization, or spreading your computation over a mini-cluster using plain ssh, or use Slurm, PBS, Sun GridEngine, you name it.
Cons: Users must be aware of the requirement.

To assess the bottom line of this approach, pros would outweigh the cons if you have a large number of interdependent script files, and your toolkit is complex and naturally has a moderate or high learning curve and/or numerous other conventions--which is true in the case of Kaldi, w.r.t data preparation and layout. The imposed requirement to cd to one directory and do everything from it could be just one of many in your case, comparatively non-burdensome.

OPTION 2. Export a variable naming the root location of all files that your scripts source.

Your script would then look like

#!/bin/bash
. "${ACME_TOOLKIT_COMMON_SCRIPTS:?}/funcs.sh" || exit
print_header "Some text"

You must ensure that this variable is defined in the environment, by hook or by crook. The :? suffix in the variable expansion makes the script end with a fatal error message if the variable is undefined or empty, and is preferred for (a) better error message and (b) a quite minor security risk of sourcing unintended code.

Pros: Still pretty readable code.
Cons: There should be an external mechanism to set the variable per installation, either per-user or machine-wide.
Cons/Meh: Slurm must be allowed to propagate your environment to the job step. This is usually so, and is on by default, but there may be cluster setups that limit the user's environment propagation to a list of administrator-approved variables.

Returning to Kaldi's example, if your workload is low, and you want to be able to parallelize to e. g. 5–10 machines on premises using ssh instead of Slurm, you'd have to either whitelist this specific environment variable in both the sshd and ssh client configurations, or make sure it is set to the same correct value on every machine.

The bottom line here in general (i. e., nothing else considered) is approximately same as that of the Option 1: one more thing to troubleshoot; possible infrastructure configuration issues, but still quite fitting for a large program with more than a dozen or two of interdependent bash scripts.

However, this option becomes more lucrative if you know you won't ever have to port your code to any other workload manager than Slurm, and even more lucrative if your WLM is one or few specific clusters, so you can rely on their unchanging configuration.

OPTION 3. Write a "launcher" script to give to sbatch to launch any command.

The launcher would take the name of the script (or any program, to that matter) to run as its first argument, and pass the rest of the arguments to the invoked script/commnd. The script can be one and same to wrap any of your scripts, and exists solely to make your sourced script discovery logic work.

The launcher script is utterly trivial:

$ cat ~/launcher
#!/bin/bash
prog=${1:?}; shift
exec "$prog" "$@"

Running the following script (from an NFS mount at /xa, naturally)

$ cat '/xa/var/tmp/foo bar/myscript.sh'
#!/bin/bash
printf 'Current dir: '; pwd
printf 'My command line:'; printf ' %q' "$0" "$@"; printf '\n'
echo "BASH_SOURCE[0]='${BASH_SOURCE[0]}'"
# The following line is the one that gave fits in your case.
my_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
echo "my_dir='$my_dir'"

with the current dir being /tmp with the sbatch command below (and testing proper quoting never hurts)

$ pwd
/tmp
$ sbatch -o /xa/var/tmp/%x-%A.out -N1 ~/launcher \
    '/xa/var/tmp/foo bar/myscript.sh' "The skies are painted with unnumber'd sparks" 1 2 '' "3 4"
Submitted batch job 19740

yields this output file:

$ cat /xa/var/tmp/launcher-19740.out
Current dir: /tmp
My command line: /xa/var/tmp/foo\ bar/myscript.sh The\ skies\ are\ painted\ with\ unnumber\'d\ sparks 1 2 '' 3\ 4
BASH_SOURCE[0]='/xa/var/tmp/foo bar/myscript.sh'
my_dir='/xa/var/tmp/foo bar'

Pros: You can run your existing script as is.
Pros: The command you give to launcher does not have to be a shell script.
Cons: And that's a big one. You cannot use #SBATCH directives in your script.

In the end, you'll likely end up writing either an individual top-level script to simply call sbatch invoking your script via this common launcher with a buttload of sbatch switches, or write a customized launcher script for each of your computing scripts, listing all the required #SBATCH directives. Not much win here.

Bottom line: if all your batch job submissions are very similar so that you can factor the absolute majority of sbatch options into #SBATCH directives in a single launcher script, this is an option to consider. Note though that all jobs will be named "launcher" unless you name them with the sbatch's -J switch, which means you either won't to be able to factor out all sbatch switches into a single file, or cope with this quite dull, at first sight, naming scheme³ and id your jobs some other way.

So, in the end, pick you poison that seems the tastiest to you, and go with it. There is no perfect solution, but there should be an acceptable way to achieve what you want.

---

¹ Of which I happen to be both an active user and a contributor.
² A test of the form . ./cmd.sh || exit would have been more robust, and should always be used, but our top-level experiment scripts are usually pretty lax, compared to core scripts.
³ But as any one of nearly 10,000,001 people in the US named Smith, Johnson, Williams, Jones, Brown or Morris "Moe" Jette can confirm, it's not necessarily a big deal.

Answer 2

You can do this by changing the working directory for your script.sh with:

sbatch -p PARTITION -t TIME -N NODES -D /full/path/to/my/ /full/path/to/my/script.sh PARAMETERS-LIST

Then in your script you can simply do source "funcs.sh"