CODEWITHSUNDEEP
pythonpython-3.xairflowairflow-scheduler
Shweta Verma
Shweta Verma

Reputation: 41

Airflow Import Issue

We have configured the Airflow and there are two airflow dags working fine. I have created a new dag and can see the same in the Airflow UI . But when I try to turn ON the Airflow Dag from UI it gives me the following error "DAG <sample_dag> seems to be missing".

I tried to troubleshoot the issue using the following ways:

  1. Firstly, I tried to run my pipeline py file to see if there are any issues with it :
     python sample_dag_pipeline.py
['/home/airflow/airflow-install/dags/ABC_Middleware', '/home/airflow/airflow-install/dags', '/home/airflow/airflow-install/sample_folder/bin/python', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/home/airflow/airflow-install/sample_folder/lib64/python3.6/site-packages', '/home/airflow/airflow-install/sample_folder/lib/python3.6/site-packages']

It seems to work fine.

  1. Then I try to run : airflow list_dags It gives me the following error:
  (env) [airflow@Airflow-VM dags]$ airflow list_dags
[2021-02-02 08:41:34,503] {__init__.py:50} INFO - Using executor LocalExecutor
[2021-02-02 08:41:34,507] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow-install/dags
['/home/airflow/airflow-install/sample_folder/bin', '/home/airflow/airflow-install/dags/AirflowMiddleware', '/home/airflow/airflow-install/sample_folder/bin/python', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/home/airflow/airflow-install/sample_folder/lib64/python3.6/site-packages', '/home/airflow/airflow-install/sample_folder/lib/python3.6/site-packages', '/home/airflow/airflow-install/dags', '/home/airflow/airflow-install/config', '/home/airflow/airflow-install/plugins']
[2021-02-02 08:41:35,044] {dagbag.py:259} ERROR - Failed to import: /home/airflow/airflow-install/dags/sample_dag_pipeline.py
Traceback (most recent call last):
 File "/home/airflow/airflow-install/sample_folder/lib64/python3.6/site-packages/airflow/models/dagbag.py", line 256, in process_file
   m = imp.load_source(mod_name, filepath)
 File "/usr/lib64/python3.6/imp.py", line 172, in load_source
   module = _load(spec)
 File "<frozen importlib._bootstrap>", line 684, in _load
 File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
 File "<frozen importlib._bootstrap_external>", line 678, in exec_module
 File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
 File "/home/airflow/airflow-install/dags/sample_dag_pipeline.py", line 8, in <module>
   from abc.def.gef import sample_dag_jobs
ImportError: cannot import name 'sample_dag_jobs'


-------------------------------------------------------------------
DAGS
-------------------------------------------------------------------
ABC_V01
DEF_V01

Lemme show my sample_dag_pipeline.py file:

(env) [airflow@Airflow-VM dags]$ cat sample_dag_pipeline.py | head -20

import sys
import traceback
from importlib import reload

sys.path.insert(0, "/home/airflow/airflow-install/dags/ABC_Middleware")

from datetime import datetime
from abc.def.gef import sample_dag_jobs

dagutils = reload(cai_jobs_pipeline)

from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator
from airflow.utils.trigger_rule import TriggerRule

DAG_NAME = "ABC_DataPipeline_V1.0"

Observation:

So in the above sample_dag_pipeline.py file I am referring to the code present in other directory(ABC_Middleware) by first adding it in sys path and then importing the modules from it. I did the same for my other two dags(ABC_V01, DEF_V01) for them it is working fine but for this one it is not.

when I run the py file it prints me the sys paths and i can see the path added in the sys.('/home/airflow/airflow-install/dags/ABC_Middleware')

But when I run : airflow list_dags in the sys paths I am not able to see the sys path of my directory ('/home/airflow/airflow-install/dags/ABC_Middleware')

I have tried many options but all seems to get stuck here only. Can anyone suggest what could have happened?

Upvotes: 4

Views: 4550

Answers (2)

Arran Duff
Arran Duff

Reputation: 1454

  • I had a similar issue. The import was definitely valid and worked when I ran python my_dag.py and also worked in a python shell.
  • A restart of the airflow server fixed the issue for me

Upvotes: 0

Ash Berlin-Taylor
Ash Berlin-Taylor

Reputation: 4048

This might be a problem due to the fact you are using abc as a package, which clashes with the built-in stdlib module of the same name, and since that is imported by Airflow before your DAG file is loaded, Python is looking under the system path for further modules.

Try re-naming abc to something else.

Upvotes: 0

Related Questions