CODEWITHSUNDEEP
filecachingbuildworkflowsnakemake
Vincent Darbot
Vincent Darbot

Reputation: 237

How to avoid running Snakemake rule after input or intermediary output file was updated

Even if the output files of a Snakemake build already exist, Snakemake wants to rerun my entire pipeline only because I have modified one of the first input or intermediary output files.

I figured this out by doing a Snakemake dry run with -n which gave the following report for updated input file:

Reason: Updated input files: input-data.csv

and this message for update intermediary files

reason: Input files updated by another job: intermediary-output.csv

How can I force Snakemake to ignore the file update?

Upvotes: 20

Views: 7360

Answers (3)

Cristian García Romero
Cristian García Romero

Reputation: 31

In case --touch (with --force, --forceall or --forcerun as the official documentation says that needs to be used in order to force the "touch" if doesn't work by itself) didn't work out as expected, ancient is not an option or it would need to modify too much from the workflow file, or you faced https://github.com/snakemake/snakemake/issues/823 (that's what happened to me when I tried --force and --force*), here is what I did to solve this solution:

  1. I noticed that there were jobs that shouldn't be running since I put files in the expected paths.
  2. I identified the input and output files of the rules that I didn't want to run.
  3. In the order of the rules that were being executed and I didn't want to, I executed touch on the input files and, after, on the output files (taking into account the order of the rules!).

That's it. Since now the timestamp is updated according the rules order and according the input and output files, snakemake will not detect any "updated" files.

This is the manual method, and I think is the last option if the methods mentioned by the rest of people don't work or they are not an option somehow.

Upvotes: 2

dariober
dariober

Reputation: 9062

In addition to Eric's answer, see also the ancient flag to ignore timestamps on input files.

Also note that the Unix command touch can be used to modify the timestamp of an existing file and make it appear older than it actually is:

touch --date='2004-12-31 12:00:00' foo.txt 
ls -l foo.txt 
-rw-rw-r-- 1 db291g db291g 0 Dec 31  2004 foo.txt

Upvotes: 13

Eric C.
Eric C.

Reputation: 3368

You can use the option --touch to mark them up to date:

--touch, -t
Touch output files (mark them up to date without really changing them) instead of running their commands. This is used to pretend that the rules were executed, in order to fool future invocations of snakemake. Fails if a file does not yet exist.

Beware that this will touch all your files and thus modify the timestamps to put them back in order.

Upvotes: 17

Related Questions