CODEWITHSUNDEEP
pythonairflow-schedulerairflow
DAC
DAC

Reputation: 41

Airflow- Using TriggerDAGRunOperator to trigger DAG of different schedule

I'm new to Airflow. I need to come up with a clean easy solution for DAG dependencies with different schedules.

I have DAG 1 running Daily and DAG 2 - Weekly. How do I use TriggerDAGRunOperator to trigger weekly DAG from Daily one?

DAG 1:

with DAG('DAG 1',
     schedule_interval='0 10 * * *'
     ) as dag:

TASK1 = BashOperator(task_id='TASK1',
                bash_command='sample')
TRG_TASK=TriggerDAGRunOperator(task_id='TRG_TASK',trigger_dag_id='DAG 2')

TASK1 >> TRG_TASK

DAG 2:

with DAG('DAG 2',
     schedule_interval='15 10 * * 5'
     ) as dag:

TASK1 = BashOperator(task_id='TASK1',
                bash_command='sample')

I know I can use ExternalTaskSensor Operator and mention timedelta, but it would become messy in long run.

Is there any easy/clean option with TriggerDAGRunOperator to check everyday if DAG 2 is indeed scheduled to run for that day then only trigger it else skip it on other days?

Thanks

Upvotes: 0

Views: 3073

Answers (1)

Josh Fell
Josh Fell

Reputation: 3589

Is DAG2 always going to be triggered by DAG1 and you only want DAG2 to run weekly or are you expecting DAG2 will be executed weekly via the cron schedule you listed or triggered by DAG1?

If the expectation is DAG2 only gets triggered by DAG1 but you only want DAG2 to execute weekly, you can use the ShortCircuitOperator in either DAG1 or DAG2 and update the schedule_interval of DAG2 to None (which means the DAG will only ever be triggered -- either manually or programmatically).

  • Option 1: In DAG1, add a ShortCircuitOperator task prior to executing the TriggerDagRunOperator which checks to see if the current day is the desired day of the week. If it is, then continue on with triggering DAG2.

  • Option 2: In DAG2, add the ShortCircuitOperator at the beginning of the workflow that performs the same day of week check. Again, if the desired day of week then continue on with the rest of the DAG.

Upvotes: 1

Related Questions