CODEWITHSUNDEEP
pythonslurmsnakemakesbatch
nicoluca
nicoluca

Reputation: 105

the jobs option does not define the number of submited jobs

The command that I use to trigger snakemake is:

snakemake --snakefile Snakefile --configfile config.json --latency-wait 60 --rerun-incomplete --keep-going --notemp --reason --use-singularity --singularity-args "--bind /somepath/folder" --jobs 400 --cores 16 --cluster-config clusterConfiguration/cluster.json --cluster "sbatch --partition={cluster.queue} --job-name={cluster.name} --cpus-per-task={cluster.nCPUs} --mem={cluster.memory} --time={cluster.maxTime} --output=\"jobs/{cluster.rulename}/slurm_%x_%A_%a.out\""

For some reason I do not get more than 20 jobs submitted at a time. Looking at the documentation, I think that I have to use:

--max-jobs-per-second
--max-status-checks-per-second

I do not understand why these options are available. If --jobs is defining the max number of submitted jobs, why should we defined the maximal number of cluster/drmaa jobs per second? What does it mean?

For the second option, what "Maximal number of job status checks per second" means?

Finally, will setting these two options to 400 enable me to have 400 jobs running at the same time? I can try it but I would like to understand before.

Thanks.

Upvotes: 0

Views: 260

Answers (1)

Chris_Rands
Chris_Rands

Reputation: 41188

If you run snakemake --help these options are all documented. The --jobs flag is synonymous for the --core flag that you are already using, this limits the number of cores (i.e. multi-processed jobs) to 16 in your case.

--max-jobs-per-second is the maximum rate at which jobs are submitted to your cluster (i.e. the impact on your scheduler). You normally don't need to specify this flag or the --max-status-checks-per-second one; the default values generally are okay.

  --cores [N], --jobs [N], -j [N]
                        Use at most N cores in parallel (default: 1). If N is
                        omitted, the limit is set to the number of available
                        cores.

  --max-jobs-per-second MAX_JOBS_PER_SECOND
                        Maximal number of cluster/drmaa jobs per second,
                        default is 10, fractions allowed.
  --max-status-checks-per-second MAX_STATUS_CHECKS_PER_SECOND
                        Maximal number of job status checks per second,
                        default is 10, fractions allowed.

Upvotes: 1

Related Questions