CODEWITHSUNDEEP
airflowairflow-2.x
user15749851
user15749851

Reputation: 23

Airflow retry of multiple tasks as a group

I have a group of tasks that should run as a unit, in the sense if any of the tasks from the group fail, the whole group should be marked as failed.

I want to be able to retry the group when it has failed.

For example, I have a DAG with these tasks:

taskA >> (taskB >> taskC) >> taskD

I want to say that (taskB >> taskC) is a group.

If either taskB or taskC fails, I want to be able to rerun the whole group (taskB >> taskC).

Upvotes: 2

Views: 2233

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 15979

This is a two parts question.

First, In Airflow downstream task can not effect upstream task. Assuming structure of:

taskA >> taskB >> taskC >> taskD

then if taskB is successful and taskC failed. it can not change the state of taskB to failed.

For Airflow>=2.5.0:

There is an option to clear a TaskGroup (see PR)

For Airflow<2.5.0:

Clearing (rerun) a TaskGroup is a feature that is not available.

Upvotes: 1

Related Questions