CODEWITHSUNDEEP
pythonairflow
Sristy Agrawal
Sristy Agrawal

Reputation: 1

Airflow cross dag dependency

I am trying to implement dependency between two dags: parent_dag and child_dag. parent dag has two dummy tasks: leave_work, cook_dinner child dag has three tasks: wait_for_dinner, have_dinner, play_with_food

wait_for_dinner uses external_task_sensor with external_dag_id as parent_dag and external_task_id as cook_dinner. start_date and schedule_interval are same for both the dags.

when parent_dag gets completed then wait_for_dinner should be succesfully exited and next task should run but its not happening in my case. (It is working fine if dags are not scheduled i.e. schedule_interval = @once) If dags are scheduled then wait_for_dinner is in running state forever. it is not listening parent_dag success.

Kindly help me with this.

code snippets:

parent_dag :

dag = DAG(
    'Parent_dag', 
    default_args=default_args, 
    start_date = convert_to_utc(1606712300), 
    schedule_interval=datetime.timedelta(minutes = 5),
    is_paused_upon_creation = False,
    catchup = False
    )

leave_work = DummyOperator(
    task_id='leave_work',
    dag=dag,
)
cook_dinner = DummyOperator(
    task_id='cook_dinner',
    dag=dag,
)

leave_work >> cook_dinner

child_dag:

dag = DAG(
    'Child_dag', 
    default_args=default_args,
    start_date = convert_to_utc(1606712300), 
    schedule_interval=datetime.timedelta(minutes=5),
    is_paused_upon_creation = False,
    catchup = False
    )

wait_for_dinner = ExternalTaskSensor(
    task_id='wait_for_dinner',
    external_dag_id='Parent_dag',
    external_task_id='cook_dinner',
    mode = "reschedule",
    timeout = 3600,
    dag = dag
)

have_dinner = DummyOperator(
    task_id='have_dinner',
    dag=dag,
)
play_with_food = DummyOperator(
    task_id='play_with_food',
    dag=dag,
)

wait_for_dinner >> have_dinner
wait_for_dinner >> play_with_food

Upvotes: 0

Views: 842

Answers (2)

Marc Lamberti
Marc Lamberti

Reputation: 821

Take a look at the new version of the TriggerDagRunOperator in Airflow 2.0, it became much easier than before.
Check out my video here https://youtu.be/8uKW0mPWmCk

Upvotes: -1

Steven Ensslen
Steven Ensslen

Reputation: 1376

When I want to start one dag from another, I use TriggerDagRunOperator in the parent, not ExternalTaskSensor in the child. TriggerDagRunOperator is more reliable.

Upvotes: 1

Related Questions