CODEWITHSUNDEEP
airflowdirected-acyclic-graphs
sowmiya vohinder
sowmiya vohinder

Reputation: 11

Dag scripts in airflow

I have written Dag scripts in airflow for the functions in my python scripts.For example In dag script I have imported the python script as "from data_extractor import *".I have built a workflow for the functions in the data_extractor.py in airflow dag script data_extractor_dag.py.But when I make any code changes in the data_extractor.py the airflow dag data_extractor_dag.py fails to load.During this scenario the airflow UI page is also not loaded.How to use airflow in a better way in this scenario.

Upvotes: 1

Views: 1026

Answers (1)

Sam
Sam

Reputation: 111

As far as I can tell, you have some library code in data_extractor.py and your dag definition in data_extractor_dag.py. When you edit code in data_extractor.py, the changes do not appear in your dag. Is this correct?

With Airflow, both the webserver and the scheduler refresh their view of the dags regularly. When it does this, it looks for python files that contain dags and imports them. This works fine and picks up changes if your code is all contained within the dag definition file. But if you edit an existing library file, or a file imported as an Airflow plugin, then you need to restart the scheduler and webserver in order to pick up on any changes.

Without restarting Airflow, you would expect to see an error if your DAG code is importing something new from a pre-existing library (as the library will not have been updated).

Upvotes: 1

Related Questions