CODEWITHSUNDEEP
dependenciestaskairflow
yee379
yee379

Reputation: 6762

airflow tasks based on previous runs

I have a DAG which basically setups up a Slack channel and invites a bunch of people to it. i don't really need to be creating the channel and adding the users at each DAG run if there has been no changes. at the same time, i need to ensure that the channels and the users already exist.

i was wondering whether there is a method by which Airflow could, say, run the task only if it has not succeeded in a previous DAG run or if it hadn't ran in the last hour or so.

i guess i could also do some kinda checkpointing (build a hash of users and pickle it to a file and compare in subsequent runs); but i was hoping that Airflow already had something in this manner.

Upvotes: 0

Views: 2188

Answers (1)

Viraj Parekh
Viraj Parekh

Reputation: 1381

There are a few ways you can handle this:

1) BranchOperator: You can define a custom condition and trigger a branch of your DAG to run based off of that. This might the easiest way, as you can just wrap your logic to check for the channel in a python callable. You can find a longer example here

2) Another way could be to set a TriggerRule. You can set a task that creates a channel only to run if an upstream task has succeeded or failed.

Upvotes: 2

Related Questions