CODEWITHSUNDEEP
bashmakefile
Parsa
Parsa

Reputation: 3236

Bash workaround, printing environment variable in comment

I am using a makefile to run a pipeline, the number of cores is set in the makefile as an environment variable.

At one point in the pipeline the makefile will execute a wrapper script which will start an LSF job array (HPC).

#!/bin/bash
#BSUB -J hybrid_job_name      # job name
#BSUB -n 32                   # number of cores in job
#BSUB -o output.%J.hybrid     # output file name

mpirun.lsf ./program_name.exe

The only problem here is that in the wrapper script the -n flag shoud be set by the 'CORES' environment variable, and not hard coded to 32. Is there anyway to work around so I can pass the CORES environment variable to the -n flag.

Upvotes: 1

Views: 284

Answers (1)

msleigh
msleigh

Reputation: 164

You could generate the wrapper script that contains the "#BSUB" directives on the fly, before submitting it to LSF. E.g. create a template such as job_script.tmpl in advance:

#!/bin/bash
#BSUB -J hybrid_job_name      # job name
#BSUB -n %CORES%              # number of cores in job
#BSUB -o output.%J.hybrid     # output file name

mpirun.lsf ./program_name.exe

and then in your makefile do:

sed s/%CORES%/${CORES}/g job_script.tmpl > job_script.lsf
bsub < job_script.lsf

Alternatively, you can set options to bsub on the command line as well as via #BSUB directives inside the job script. So in the makefile do:

bsub -n $CORES < job_script.lsf

The value passed to bsub on the command line will override the value defined by the #BSUB -n directive inside the job script. This way is a bit simpler but the first way has the benefit of recording the number of cores used in the job, for future reference.

Upvotes: 2

Related Questions