CODEWITHSUNDEEP
pythonairflow
A H
A H

Reputation: 2570

Retry a different task in a dag

I have an airflow DAG that looks like this

(A1 -> B1) -> C1 -> D1
(A2 -> B2) -> C2 -> D2
(A3 -> B3) -> C3 -> D3

The problem is:

What is the best way to do that?

I tried branching back to A, e.g.

But I get the error Cycle detected in DAG

Upvotes: 0

Views: 200

Answers (2)

Jorrick Sleijster
Jorrick Sleijster

Reputation: 1156

There are multiple parts of this question that we should consider:

  1. When B fails, why would we want to restart A? Is it because B modifies something when it fails? In that case, wouldn't it make more sense to write the new data somewhere else instead of overwriting?
  2. In the case the output of A influences whether B succeeds, we should verify the output of task A, inside task A itself.
  3. If it really is the case that A needs to be re-done when B fails and we can't prevent this, then it would be best to unify the two. You have two ways of doing this:

A. You can create a custom operator that upon execute() calls A first and then B.

B. You can create a PythonOperator instance and use Airflow hooks or pure python code to get the job done.

Upvotes: 2

AssertionError
AssertionError

Reputation: 346

DAG is for "Directed Acyclic Graph", the important part is "Acyclic", you can't perform a cycle, your only option is to relaunch DAG when it fails. Or only one task if you can isolate it and made it idempotent.

This answer may help you

Upvotes: 1

Related Questions