CODEWITHSUNDEEP
airflow
kms333
kms333

Reputation: 3257

How to get Airflow's previous execution date regardless of how the DAG is triggered?

When I trigger a DAG manually, prev_execution_date and execution_date are the same.

echo_exec_date = BashOperator(
    task_id='bash_script',
    bash_command='echo "prev_exec_date={{ prev_execution_date }} execution_date={{ execution_date }}"',
    dag=dag)

results in:

prev_exec_date=2022-06-29T08:50:37.506898+00:00 execution_date=2022-06-29T08:50:37.506898+00:00

They are different if the DAG is triggered automatically by the scheduler.

I would like to have prev_execution_date regardless of triggering it manually or automatically.

Upvotes: 3

Views: 3318

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 15931

When manually triggering DAG, the schedule will be ignored, and prev_execution_date == next_execution_date == execution_date This is explained in the Airflow docs

This is because previous / next of manual run is not something that is well defined. Consider you have a daily schedule (say at 00:00) and you invoke a manual run on 13:00. What is the expected next schedule? should it be daily from 00:00 or daily from 13:00? a DagRun can have only 1 prev and only 1 next. In your senario it seems like you are interested in a case where there can be more than 1 or that the manual run "comes between" the two scheduled runs. This is not something that Airflow supports - It really over complicate things.

If you want to workaround it you can create custom macro that checks the run_type, searches the specific DagRun that you consider as previous and return it's execution_date. Be noted that it might create some side effects (overlapping data interval process etc..) you need to really verify that the logic you implement make sense for your specific use case.

Upvotes: 5

Related Questions