CODEWITHSUNDEEP
snakemake
Matt Miller
Matt Miller

Reputation: 1

Snakemake cluster mode with different CPU count for each rule

I'm working with snakemake on a shared compute space with the SLURM scheduler. Most of my rules require very few resources/CPUs, while a couple require a lot. I want to setup snakemake to use minimal resources for every rule except the rules that requires lots of resources. From my testing, snakemake will always use the core counts and memory requests in the config.yaml when submitting each rule as a subjob in cluster mode. When I define threads and mem_mb in each rule, it doesn't consider those values when submitting the rule as a job. Is there a way in cluster mode to have snakemake look at the rule's threads/mem_mb when submitting a job, or some way to customize the SLURM resource requests based on each rule? Below are my config.yaml (I call it cluster.yml in the code) and the line that runs snakemake. Thank you!

__default__:
  partition: 'standard'
  group: 'mygroup'
  M: '[email protected]'
  time: '24:00:00'
  n: "2"
  m: '8gb'
  o: 'out/{rule}.{wildcards}.out'
  e: 'err/{rule}.{wildcards}.err'
  N: "1"

snakemake --cluster "sbatch -A {cluster.group} -p {cluster.partition} -n {cluster.n} -t {cluster.time} -N {cluster.N} --mem={cluster.m} -e {cluster.e} -o {cluster.o}"  --cluster-config config/cluster.yml -j 30 --latency-wait 30

Upvotes: 0

Views: 364

Answers (1)

Matt Miller
Matt Miller

Reputation: 1

Actually I think I figured it out thanks to here. Doing

n: "{threads}"
m: "{resources[mem_mb]}mb"

will use the cores/memory defined in each rule.

Upvotes: 0

Related Questions