CODEWITHSUNDEEP
pythonairflowairflow-scheduler
prasanna Kumar
prasanna Kumar

Reputation: 291

Airflow ExternalTaskSensors - to create dags dependency

I would like to create dag dependency on DAG A and DAG B. DAG A has two tasks: TASK1 and TASK2 . DAG B has 3 tasks: TASK1, TASK2 and TASK3.

My requirment is for DAG B to start after DAG A TASK1.

Both DAGS are runs in hourly, DAG A runs @every hours EX: 10.00 and DAG B runs @every hour ex:10.30.

I'm using Airflow and the operator EXternalTaskSensors but it is not working.

external_dag_id='DAG A',
external_task_id='TASK1',
allowed_states=None,
execution_delta=None,
execution_date_fn=None,

Upvotes: 2

Views: 4517

Answers (1)

Chengzhi
Chengzhi

Reputation: 2591

If you check execution_delta there, you have None, the documentation[https://github.com/apache/incubator-airflow/blob/master/airflow/operators/sensors.py#L194] said:

:param execution_delta:time difference with the previous execution to look at, the default is the same execution_date as the current task. For yesterday, use [positive!] datetime.timedelta(days=1). Either execution_delta or execution_date_fn can be passed to ExternalTaskSensor, but not both.

Short answer is, since you are running DAG A and DAG B at different time, you need to put the execution_delta, otherwise, it assume your other DAG runs at the same time, in that case, no DAG run found and you may get something unexpected. So try something like datetime.timedelta(minutes=30)

Upvotes: 2

Related Questions