CODEWITHSUNDEEP
pythonmultithreadingconfigurationconfigsnakemake
epigenome
epigenome

Reputation: 81

Snakemake multiple Config Files?

Quick question regarding snakemake and using more than one config file.

I'm creating a rather large pipeline for genomic preProcessing, and the idea is that it is user-friendly and accounts for the lowest-common-denominator in terms of user ability.

So, instead of allowing the user to define in the main configfile, the threads used by certain tools. I want to implement snakemake's "workflow.cores * percentage" capabilities, that is; I can for each tool specify a percentage of the total cores (as defined in the command line with --cores #).

This keeps things simple and doesn't confuse the user with all the options in the main config file. But I still want to allow for the user to fine-tune the number of threads if they so please. The idea is I have a main config file without the thread numbers, and a second config file with the thread numbers.

If the user decides they want to use their own decided number of threads, they will simply (in the main config) provide an affirmative in a key that's something like: manualThreadChoice: "yes"

And have a simple if statement in the Snakefile that will, if the manualThreadChoice is affirmative, then provide the number of threads to the respective rules accordingly, otherwise just use the automatically defined thread percentages.

According to this question on biostars: https://github.com/yanailab/celseq2/issues/33

There is now the ability for defining more than one config file.

See comment:

"Thank you for reporting this issue. This is because snakemake updates their API which supports multiple config files now. Based on their log, the configfile was changed to configfiles. See: snakemake/snakemake@23624ee#diff-88e96378bf2405c8a8f8ac971519039e."

So instead of calling our config file with

configfile: "path/to/config.yaml"

we can use:

configfiles:

My question is, do we then provide two separate paths as part of a list of files: i.e.

configfiles: ["path/to/config1.yaml", "path/to/config2.yaml"]

And how do we then access key from our different configs. Since with single config files we would use:

config['key']

I've tried to access the different config files with indexing:

config[0]['key']

but this doesn't work.

I'm using Snakemake 5.7.0 which I believe is a version that has this multiple config files ability.

Upvotes: 3

Views: 2016

Answers (1)

Egelbets
Egelbets

Reputation: 336

You can just specify multiple config files at the top of your Snakefile like so:

configfile: "path/to/config1.yaml"
configfile: "path/to/config2.yaml"

Snakemake then merges all the config files together into 1 config dict that is accessible via the global variable config

Upvotes: 8

Related Questions