CODEWITHSUNDEEP
pythonairflow
bodihex
bodihex

Reputation: 89

Airflow force re-run of upstream task when cleared even though downstream if marked success

I have tasks A -> B -> C in Airflow and when I run the DAG and all complete with success, I'd like to be able to clear B alone (while leaving C marked as success). B clears and gets put into the 'no_status' state but then when I try to re-run B, nothing happens. I've tried --ignore_dependencies, --ignore_depends_on_past and --force but to no avail. B seems to only re-run if C is also cleared and then everything re-runs as expected.

The reason why I'd like to be able re-run B specifically without changing the pipeline is that some of B's external inputs may change slightly (file changed, or tweak) and I'd like to run it and evaluate it's output before restarting the downstream tasks (to mitigate any potential interruption).

Upvotes: 2

Views: 2831

Answers (2)

Pierre CORBEL
Pierre CORBEL

Reputation: 733

In the UI, when you clear a instance task, the downstream case is checked by default.

If you unchecked it, it will clear only this one and not re-run the downstream tasks

Upvotes: 0

Daniel Huang
Daniel Huang

Reputation: 6548

If you clear via the command line, you can specify which tasks you want cleared.

airflow clear your_dag_id --task_regex B --start_date 2017-09-25T17:00:00 --end_date 2017-09-25T17:00:00

If you want to do this via the UI, you can go to Browse -> Task Instances to find the task instance you want to clear. Deleting from there should only clear the tasks you selected. You will also need to go over to Browse -> DAG Run and set the relevant DAG run to the running state for your cleared task to be picked up.

Upvotes: 2

Related Questions