CODEWITHSUNDEEP
airflowgoogle-cloud-composer
mniehoff
mniehoff

Reputation: 517

Wait for triggered DAG

I have one control dag which triggers two other dags. The two dags should run sequentially and not in parallel. I tried solving the problem like this:

TriggerDag (using BashOp) -> ExternalDagSensor -> TriggerDag (using BashOp) -> ExternalDagSensor.

My problem is that the triggered DAG does get a specific execution_date (specific down to the seconds, not 00:00 for minutes and seconds). The DagSensor now uses the execution_time of the control dag to poke for the dependent dag and so the sensor never gets triggered, as the dependent dag has a different execution_time.

My questions:

  1. Is the Trigger->Sensor->Trigger->Sensor pattern the right way to trigger DAGs sequentially?

  2. If yes: How do I get

    a) either the execution_date of the dependent DAG after it has been triggered by the controller DAG (can then be passed to the sensor as argument)

    or

    b) the execution_date of the dependent DAG to be the same as the control DAG

If possible I do not want to query the metadata database to get the execution_time of the dependent DAG run.

Upvotes: 1

Views: 1683

Answers (1)

Wilson
Wilson

Reputation: 614

There are a couple options that might be a bit simpler.

  1. Can you combine the DAGs into a single DAG either by merging their tasks or composing them with SubDagOperator?
  2. If you really must have the DAGs be separate, try eliminating the control DAG, putting both DAGs on the same start_date and schedule_interval, and having the second DAG use an ExternalTasksSensor as its first task. Since the DAGs are on the same schedule, the execution_date of the dependent DAG will be the same as that of the dependee.

Upvotes: 2

Related Questions