Reputation: 6762
I have a DAG which basically setups up a Slack channel and invites a bunch of people to it. i don't really need to be creating the channel and adding the users at each DAG run if there has been no changes. at the same time, i need to ensure that the channels and the users already exist.
i was wondering whether there is a method by which Airflow could, say, run the task only if it has not succeeded in a previous DAG run or if it hadn't ran in the last hour or so.
i guess i could also do some kinda checkpointing (build a hash of users and pickle it to a file and compare in subsequent runs); but i was hoping that Airflow already had something in this manner.
Upvotes: 0
Views: 2188
Reputation: 1381
There are a few ways you can handle this:
1) BranchOperator: You can define a custom condition and trigger a branch of your DAG to run based off of that. This might the easiest way, as you can just wrap your logic to check for the channel in a python callable. You can find a longer example here
2) Another way could be to set a TriggerRule. You can set a task that creates a channel only to run if an upstream task has succeeded or failed.
Upvotes: 2