CODEWITHSUNDEEP
pythonairflow
user13
user13

Reputation: 391

Copy Airflow project into Airflow DAGs directory

I have an Airflow project with the following structure

my_project
 |
dags-custom-|custom_operator1.py
 |          |custom_operator2.py
 |          |custom_hook1.py
 |
my_dag.py

Normally, if I have only my_dag.py I would do

cp my_dag.py ~/airflow/dags/

However, in this case I am not sure what should I do to load the project into Airflow DAGs list?

should I do cp ~/my_project/ ~/airflow/dags/?

Upvotes: 0

Views: 840

Answers (2)

scr
scr

Reputation: 962

Airflow automatically adds ~/airflow/dags/ to the python path yes.

But you can do so manually with any folder:

export PYTHONPATH=~/my_project

This should give all of your DAGs access to the my_project packages.

Upvotes: 0

Pbd
Pbd

Reputation: 1319

Airflow tries to parse every python file as a DAG and if it finds that a python file has the DAG context, then it is loaded. In your case, you are correct, just copy over the project to airflow DAGs.

Assuming that you are using the docker image to run airflow, it would be advisable to mount your project's root into the airflow DAG root like this:

./my_project/:/opt/airflow/dags

Upvotes: 1

Related Questions