CODEWITHSUNDEEP
dependenciesexternaldirected-acyclic-graphs
Crubal Chenxi Li
Crubal Chenxi Li

Reputation: 313

Hourly DAG ExternalTaskSensor check external DAG always in "reschedule" mode

I have an hourly DAG ("dag_b") depend on the complete of an external DAG ("dag_a", also an hourly DAG that started 10 minutes earlier than "dag_b", mentioned in 'execution_delta'), hence want to use ExternalTaskSensor I've tried below where 'check_task_a' is a step of "dag_b" that checks on "dag_a" ('dag_complete' is the last step of 'dag_a')

check_task_a = ExternalTaskSensor(
        task_id="check_task_a",
        external_dag_id="dag_a",
        external_task_id="dag_complete",
        allowed_states=["success"],
        check_existence=True,
        execution_delta=timedelta(minutes=10),
        timeout=600,
        poke_interval=180,
        mode="reschedule",
        dag=dag,
    )

Both DAGs are hourly DAG, and "dag_a" has complete but the 'check_task_a' step is at 'up_for_reschedule' and then 'failed' after the timeout.

Curious anything I missed? Or other parameters I shall put in? Thanks!

Upvotes: 1

Views: 27

Answers (1)

Crubal Chenxi Li
Crubal Chenxi Li

Reputation: 313

This is resolved by removing check_existence=True, Actually it caused an error due to I manually triggered that in dev environment. While waiting for its planned schedule of automatic run, can see 'check_task_a' step is 'success'.

Upvotes: 0

Related Questions