CODEWITHSUNDEEP
airflowairflow-scheduler
Sandesh
Sandesh

Reputation: 438

apache airflow external task sensor on hourly running task

In apache airflow is it possible to add external task sensor to a hourly running job from daily job.

We have a scenario where in a daily job we need some columns which are updated by the daily job.

But some times hourly job running at 23:00 is not completing before the daily job starts.

In this case we are missing some of the update made by hourly job.

Can we add dependency on hourly job from daily job?

Upvotes: 1

Views: 1773

Answers (1)

pcc
pcc

Reputation: 61

You can create a file (i.e. SUCCESS) at the end of your hourly DAG (delete SUCCESS as the first task of this dag) and add a file sensor task in your daily job.

In this case, as long as your hourly task running, your daily job cannot be executed.

Upvotes: 2

Related Questions