CODEWITHSUNDEEP
airflow-schedulerairflow
impstuffsforcse
impstuffsforcse

Reputation: 129

DAG schedule in Airflow 2.0

How to schedule a DAG in Airflow 2.0 so that it does not run on holidays?

Question 1 : Runs on every 5th working day of the month? Question 2 : Runs on 5th working day of the month, if 5th day is holiday then it should run on next day which is not holiday?

Upvotes: 2

Views: 427

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 16139

For the moment this can't be done (at least not natively). Airflow DAGs accept either single cron expression or a timedelta. If you can't say the desired scheduling logic with one of them then you can not have this scheduling in Airflow. The good news is that Airflow has AIP-39 Richer scheduler_interval to address it and provide more scheduling capabilities in future versions.

That said, you can workaround this by setting you DAG to run with schedule_interval="@daily" and place BranchPythonOperator as the first task of the DAG. In the Python callable you can write the desired logic of your scheduling meaning that your function will return True if it's the 5th working day of the month otherwise will return False and you workflow will branch accordingly. For True it will continue to executed and for False it will end. This is not ideal but it will work. A possible template can be:

def check_fifth():
    #write the logic of finding if today is the 5th working day of the month

    if logic:
        return "continue_task"
    else:
        return "stop_task"

with DAG(dag_id='stackoverflow',
         default_args=default_args,
         schedule_interval="@daily",
         catchup=False
         ) as dag:

    start_op = BranchPythonOperator(
        task_id='choose',
        python_callable=check_fifth
    )

    stop_op = DummyOperator(
        task_id='stop_task'
    )

    #replace with the operator that you want to actually execute
    continue_op = YourOperator(
        task_id='continue_task'
    )

    start_op >> [stop_op, continue_op]

Upvotes: 2

Related Questions