CODEWITHSUNDEEP
pythonpython-2.7ubuntuairflow
Sunny Duckling
Sunny Duckling

Reputation: 357

Where should I create a folder with my own DAGs for Airflow?

I have to work with Airflow on Windows. I'm new to it, so I have a lot of issues.

So, I've already done all the steps from one of the tutorial using Ubuntu:

sudo apt-get install software-properties-common
sudo apt-add-repository universe
sudo apt-get update
sudo apt-get install python-pip
export SLUGIFY_USES_TEXT_UNIDECODE=yes
pip install apache-airflow

Python version: Python2.7.15+.

I have my /home/user_name/ where I created AirflowHome folder. Then I did:

nano ~/.bashrc
export AIRFLOW_HOME=~/home/user_name/AirflowHome (#wrote on the last string)

I ran

airflow initdb

in /home/user_name/.

Should have done this instead in /home/user_name/AirflowHome/ ?

I also ran:

airflow webserver -p 8080

It never finished (is it alright?), but it seemed to work as I was able to reach primary Airflow window.

I've already tried putting a 'dags' folder (with my own dags) to 

/home/user_name/airflow/
/home/user_name/AirflowHome/
/home/user_name/.local/lib/python2.7/site-packages/airflow/dag/

None of these showed my SampleFile.py on Airflow webserver (I checked dag_id in the file, it is alright).

Now I need to understand where I can create a 'dags' folder where I would put all of my DAGs.

I would be very grateful, if you helped me fix it.

Upvotes: 10

Views: 29283

Answers (2)

Sunny Duckling
Sunny Duckling

Reputation: 357

I solved the problem.

In airflow.cfg I put dags_folder=/home/user_name/.local/lib/python2.7/site-packages/airflow/example_dags .

It worked.

When you need to add a file, put it in the folder and run airflow initdb in Ubuntu. When you need to remove a file from this folder, delete the file and then sign out and sign in to Windows, then run airflow db init and airflow webserver in Ubuntu.

Upvotes: 3

amoskaliov
amoskaliov

Reputation: 799

Correct path to your DAG folder is set in airflow.cfg file (it's located at /home/user_name/AirflowHome/airflow.cfg). Check the dags_folder parameter (the first one actually).

It never finished (is it alright?)

Yes, it's ok. You can add -D parameter to start it in background.

BTW, as I remember, you should also run airflow scheduler in another terminal.

Upvotes: 6

Related Questions