CODEWITHSUNDEEP
airflow
dinigo
dinigo

Reputation: 7438

Airflow catchup new tasks added to dag

When you deploy a new DAG with catchup=True Airflow scheduler automatically schedules all the tasks from the start_date till now. It finishes and you get a lot of dark green balls meaning those DAGs in the past have been run.

But now you want to add new tasks to the past DAG runs. What should I do to make the old DAG runs recognize the new tasks and run them?

Upvotes: 1

Views: 2959

Answers (2)

Marconius
Marconius

Reputation: 731

If you have access to airflow cli, you can use the backfill command to run the new task for any period of time. The command does not re-run any of the tasks that have already run in subsequent dag runs.

Start and end dates to backfill are optional, so you could just run

$ airflow backfill your_dag_id

Upvotes: 1

SergiyKolesnikov
SergiyKolesnikov

Reputation: 7815

You should clear the old DAG runs (can also be done in the GUI). The cleared runs will be rescheduled including the newly added tasks.

Upvotes: 2

Related Questions