CODEWITHSUNDEEP
pythonscipydistributed-computingpipeline
user248237
user248237

Reputation:

decorators to make a sequential set of tasks in Python?

I am writing a program that executes a set of tasks sequentially. Each task is a function that outputs a new file, but any given task should not execute if the filename already exists. I find myself writing this kind of code over and over again:

task1_fname = task1()
# come up with filename for next task
task2_fname = "task2.txt"
if not os.path.isfile(task2_fname):
  # task2 might depend on task1's output, so it gets task1's filename
  task2(task1_fname)
task3_fname = "task3.txt"
if not os.path.isfile(task3_fname):
  task3(...)

The basic idea is that if a file is present (and ideally nonempty) then you should not execute the task that generates this file.

What's the best Pythonic way to express this without having to write os.path.isfile calls everytime? Could decorators express this more concisely? Or something along the lines of:

with task2(task1_fname):
  # continue to next task

any ideas?

Upvotes: 0

Views: 104

Answers (1)

georg
georg

Reputation: 215019

Are you looking for something like this?

def preserve_output(f):
    def wrap(input, output):
        if not os.path.isfile(output):
            f(input, output)
    return wrap

@preserve_output
def task1(input, output):
    ...

@preserve_output
def task2(input, output):
    ...

task1('input', 'output_1')
task2('output_1', 'output_2')
task3('output_2', 'output_3') etc

Upvotes: 3

Related Questions