CODEWITHSUNDEEP
pythonairflow
hotchocolate
hotchocolate

Reputation: 475

Importing local module (python script) in Airflow DAG

I'm trying to import a local module (a python script) to my DAG.

Directory structure: 

airflow/
├── dag
│   ├── __init__.py
│   └── my_DAG.py
└── script
    └── subfolder
        ├── __init__.py
        └── local_module.py

Sample code in my_DAG.py:

#trying to import from local module
from script.subfolder import local_module  

#calling a function in local_module.py  
a = some_function()

I get an error in Airflow saying 'Broken DAG: my_DAG. No module named 'local_module'.

I've updated Airflow to 1.9.0 but this doesn't fix the issue.

Thanks.

Upvotes: 45

Views: 51295

Answers (3)

RafaelJan
RafaelJan

Reputation: 3608

If you run Airlow in a docker then you need to do it as following:

  1. Create a folder for your modules in dags folder. For example programs
  2. Use it as following (this is the correct path for docker):
import sys
sys.path.append('/opt/airflow/dags/programs/my_module')
import my_module
task1 = PythonOperator(
        task_id='my_task_name',
        python_callable=my_module.my_func,
        dag=dag,
    )

Upvotes: 4

Shaby
Shaby

Reputation: 99

The way I do it is as following:

  1. create a Python script in your sub-folder with a main() function.
  2. in your dag file include a path declaration for the sub-folder and the file

Now you can use this script in your PythonOperator

import sys
sys.path.insert(0,"/root/airflow/dags/subfolder"))
import subfolder.script_name as script
...    
t1=PythonOperator(
    task_id='python_script',
    python_callable=script.main,
    dag=dag
)

Upvotes: 5

tobi6
tobi6

Reputation: 8239

This usually has to do with how Airflow is configured.

In airflow.cfg, make sure the path in airflow_home is correctly set to the path the Airflow directory strucure is in.

Then Airflow scans all subfolders and populates them so that modules can be found.

Otherwise, just make sure the folder you are trying to import is in the Python path: How to use PYTHONPATH

Upvotes: 12

Related Questions