CODEWITHSUNDEEP
airflow
Codious-JR
Codious-JR

Reputation: 1748

Airflow Packaged Dag (Zip) not recognized

I am trying to package my Repository with my Dag in a Zip file like it states here in the documentation. So i have followed the convention in the documentation, which is to keep the dag in the root of the zip, and the sub directories are viewed as packages by airflow.

My zip file has the following contents:

$ unzip -l $AIRFLOW_HOME/dags/test_with_zip.zip 
Archive:  /home/arjunc/Tutorials/airflow/dags/test_with_zip.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2018-03-29 17:46   helloworld/
      189  2018-03-29 17:22   helloworld/hello.py
        0  2018-03-29 17:18   helloworld/__init__.py
      461  2018-03-29 17:24   test_with_zip_dag.py
---------                     -------
      650                     4 files

Where test_with_zip_dag.py is the file in the root directory with the Dag definitions as follows:

from datetime import datetime

from airflow import DAG
from airflow.operators.python_operator import PythonOperator

from helloworld.hello import HelloWorld


def run():
    return HelloWorld().run()


dag = DAG('test_with_zip', description='Test Dependencies With Zipping',
          schedule_interval='0 12 * * *',
          start_date=datetime(2017, 3, 20), catchup=False)

hello_operator = PythonOperator(task_id='hello_task', python_callable=run, dag=dag)

I have placed this zip in the default dags directory $AIRFLOW_HOME/dags, but my dag isn't recognized!

What am I doing wrong?

Update

When I restarted the webserver, the task test_with_zip has popped up, but it is not runnable because the Scheduler doesn't seem to recognize it. I get the following error for it (from the web interface):

This DAG seems to be existing only locally. The master scheduler doesn't seem to be aware of its existence.

Upvotes: 1

Views: 6092

Answers (3)

marioquark
marioquark

Reputation: 427

The DAG python file has to be in the root of the zip package. See https://airflow.apache.org/docs/stable/concepts.html#packaged-dags

Upvotes: 1

Ayush Chauhan
Ayush Chauhan

Reputation: 459

Which version of airflow are you on? Airflow 1.8.1 had problems with loading dags from zips. This issue was fixed in 1.8.3. https://issues.apache.org/jira/browse/AIRFLOW-1357

I recommend that you should update to the latest version of Airflow ie 1.9.0

Upvotes: 3

tobi6
tobi6

Reputation: 8239

You mention only to restart the webserver.

You also need to start the scheduler with airflow scheduler.

Also, see more steps to check here: Airflow 1.9.0 is queuing but not launching tasks

Upvotes: 0

Related Questions