CODEWITHSUNDEEP
airflowairflow-scheduler
Joaquín L. Robles
Joaquín L. Robles

Reputation: 6504

Start @daily Airflow dynamic DAGs right after creation

I'm creating dynamic DAGs and want them to be triggered inmediately after they are created, so I tought setting catchup=True and start_date as the DAG's creation date minus one frequency using Croniter.

This is my code so far, it works for @hourly DAGs but not for @daily as they are created but not triggered even they should:

    # get the start date to create the dag
    created_at = pendulum.parse('2022-04-12T16:04:42.49305-03:00')
    freq = '@daily'

    # get the cron expression of the DAG scheduling and iterate one previous frequency
    cron = croniter(freq, created_at)
    cron.get_prev(datetime) 
    # > datetime.datetime(2022, 4, 12, 0, 0, tzinfo=Timezone('-03:00'))

    # create the DAG with start_date as the connection creation date
    # minus 1 scheduling frequency so it is triggered right after its 
    # creation (because of catchup=True)
    default_args = {
      'start_date': cron.get_prev(datetime),
      # > datetime.datetime(2022, 4, 11, 0, 0, tzinfo=Timezone('-03:00'))
      'retries': 1,
      'retry_delay': timedelta(minutes=5),
      'email_on_failure': False,
      'email_on_retry': False
    }

The DAG is created with 'start_date': DateTime(2022, 4, 11, 3, 0, 0, tzinfo=Timezone('UTC')) but this is when the next run scheduled:

Next DAG run

How can I get it run right away?

Upvotes: 0

Views: 257

Answers (1)

hopeIsTheonlyWeapon
hopeIsTheonlyWeapon

Reputation: 567

Please check the below from

https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html?highlight=schedule%20interval

Similarly, since the start_date argument for the DAG and its tasks points to the same logical date, it marks the start of the DAG's first data interval, not when tasks in the DAG will start running. In other words, a DAG run will only be scheduled one interval after start_date

Can you adjust your start_date accordingly with catchup=True and see what happens ?

Upvotes: 1

Related Questions