CODEWITHSUNDEEP
airflow
lampShadesDrifter
lampShadesDrifter

Reputation: 4139

When does a airflow dag definition get evaluated?

Suppose I have an airflow dag file that creates a graph like so...

def get_current_info(filename)
    current_info = {}
    <fill in info in current_info relevant for today's date for given file>
    return current_info

files = [
    get_current_info("file_001"),
    get_current_info("file_002"),
    ....
]

for f in files:
    <some BashOperator bo1 using f's current info dict>
    <some BashOperator bo2 using f's current info dict>
    ....

    bo1 >> bo2
    ....

Since these values in the current_info dict that is used to define the dag changes periodically (here, daily), I would like to know by what process / schedule the dag definition gets updated. (I print the current_info values each run and values appear to be updating, but curious as to how and when exactly this happens).

When does a airflow dag definition get evaluated? referenced anywhere in the docs?

Upvotes: 1

Views: 762

Answers (2)

lampShadesDrifter
lampShadesDrifter

Reputation: 4139

After some discussion on the [airflow email list][1], it turns out that airflow builds the dag for each task when it is run (so each tasks includes the overhead of building the dag again (which in my case was very significant)).

See more details on this here: https://stackoverflow.com/a/59995882/8236733

Upvotes: 2

GTO
GTO

Reputation: 56

The DAGs are evaluated in every run of the scheduler.

This article describes how the scheduler works and at what stage the DAG files are picked up for evaluation.

Upvotes: 1

Related Questions