CODEWITHSUNDEEP
airflow
partlov
partlov

Reputation: 14277

New task in DAG blocks further DAG executions

We have an ETL DAG which is executed daily. DAG and tasks have the following parameters:

When we add a new task, due to depends_on_past property, no new DAG runs get scheduled, as all previous states for new task are missing. We would like to avoid having to run manual backfill or manually marking previous runs from UI as it can be easily forgotten, and we also have some dynamic DAGs where tasks get added automatically and halt future DAG executions.

Is there a way to automatically set past executions for new tasks as skipped by default, or some other approach that will allow future DAG runs to execute without human intervention? We also considered creating a maintenance DAG that would insert missing task executions with skipped state, but would rather not go this route. Are we missing something as the flow looks like a common thing to do?

Upvotes: 2

Views: 267

Answers (2)

korhner
korhner

Reputation: 733

A hacky solution is to set depends_on_past to False as max_active_runs=1 will implicitly guarantee the same behavior. As of the current Airflow version, the scheduler orders both dag runs and task instances by execution date before running them (checked 1.10.x but also 2.0)

Another difference is that next execution will be scheduled even if previous fails. We solved this by retrying unlimited times (setting a ver large number), and alert if retry number is larger than some value.

Upvotes: 1

Alan Ma
Alan Ma

Reputation: 591

Defined in Airflow documentation on BaseOperator:

depends_on_past (bool) – when set to true, task instances will run sequentially and only if the previous instance has succeeded or has been skipped. The task instance for the start_date is allowed to run.

As long as there exists a previous instance of the task, if that previous instance is not in the success state, the current instance of the task cannot run. When adding a task to a DAG with existing dagrun, Airflow will create the missing task instances in the None state for all dagruns. Unfortunately, it is not possible to set the default state of task instances.

I do not believe there is a way to allow future task instances of a DAG with existing dagruns to run without human intervention. Personally, for depends_on_past enabled tasks, I will mark the previous task instance as success either through the CLI or the Airflow UI.

Looks like there is an Github Issue describing exactly what you are experiencing! Feel free to bump this PR or take a stab at it if you would like.

Upvotes: 2

Related Questions