CODEWITHSUNDEEP
loggingsnakemake
soungalo
soungalo

Reputation: 1338

snakemake - dynamically set log for each rule

I'd like to set the log file (log directive) for each of the workflow rules dynamically, so it contains the rule name. I tried to use the set_log() function like I do with set_params(), but am getting a cryptic error. Here is what I tried:

logs_dir = config['logs_dir']

rule1:
    ...

rule2:
    ...

for r in workflow.rules:
    r.set_log(os.path.join(logs_dir, r.name + '.log'))

and getting: AssertionError in file ... in line ...
I am probably misusing set_log, but couldn't figure it out. Any ideas?

UPDATE
Following the suggestions in the answers, I tried the following:

rule all:
    input:
        'b'

rule myrule:
    input:
        'a'
    output:
        'b'
    log:
        dummy = 'dummy.log'
    shell:
        """
        cp {input} {output}
        """

for r in workflow.rules:
    r.set_log(real=r.name + '.log')

Unfortunately, running it with snakemake -s snakefile -j 1 resulted again in an unspecified Assertion Error.

Upvotes: 4

Views: 361

Answers (2)

euronion
euronion

Reputation: 1277

Based on Ulthran's answer you could also set a dummy log (with the same name) for all rules and then simply add the "real" log dynamically via loop:

rule rule_name:
    ...
    log:
        dummy="logs/will-be-overwritten.txt",
    shell:
        """
        echo 'fake log' > {log[0]}
        echo 'real log' > {log.real}
        """


for r in workflow.rules:
    print(r.name)
    r.set_log(real="logs/" + r.name + ".log")

Note that this will create problems if you rule contains wildcards: If you do not handle the wildcards in your dynamically-added-log, then I don't know what will happen. Either Snakemake will throw an error or it will use the same logfile for all instances of the rule. I can only speculate that the potential existance of wildcards is one reason for the assertion error.

Upvotes: 0

Ulthran
Ulthran

Reputation: 308

The set_log function is defined here and I think it eventually runs into this assertion that throws this very-not-informative AssertionError. The way to fix it is to add an inline log definition to each rule:

logs_dir = config['logs_dir']

rule1:
    log: "rule1.log"
    ...

rule2:
    log: "rule2.log"
    ...

for r in workflow.rules:
    r.set_log(os.path.join(logs_dir, r.name + '.log'))

which unfortunately defeats the purpose of looping through and setting logs (if I'm understanding what you want to do here correctly). I recently ran into a very similar issue with setting threads without defining a threads value in each rule. Kind of annoying coming at it from this perspective but I think there is good reason behind this behavior.

One way you might be able to still avoid writing log: 'rule_name.log' in every rule would be using rule inheritance although I'm not totally sure how passing the names and wildcards would work. If you get something like this working please let me know.

Upvotes: 2

Related Questions