CODEWITHSUNDEEP
airflowairflow-2.xairflow-webserver
Salva
Salva

Reputation: 113

error while importing DAG file in Airflow 2.5.0

when I start my airflow schedular and webserver my bigdata.py file not getting imported, below is the error which am getting.

​Broken DAG: [/home/adminn/airflow/dags/bigdata.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'airflow.providers.apache'

this is the DAG I have written, am I missing something?

here I am trying to pull MySQL table using sqoop and load in HDFS, and scheduling this operation using Airflow.

from airflow.models import DAG
from airflow.contrib.operators.sqoop_operator import SqoopOperator
from airflow.utils.dates import days_ago


Dag_Sqoop_Import = DAG(dag_id="SqoopImport",
                      schedule_interval="* * * * *",
                      start_date=days_ago(2))

sqoop_mysql_import = SqoopOperator(conn_id="sqoop_local",
                                  table="shipmethod",
                                  cmd_type="import",
                                  target_dir="/airflow_sqoopImport",
                                  num_mappers=1,
                                  task_id="SQOOP_Import",
                                  dag=Dag_Sqoop_Import)

sqoop_mysql_import

replies appreciated,thanks.

Upvotes: 0

Views: 836

Answers (1)

Mattia
Mattia

Reputation: 1139

From airflow 2.x the providers are no longer included by default, but you have to install and then import them and the import path is changed

In your case you have to:

  • install the sqoop provider by running: pip install 'apache-airflow-providers-apache-sqoop'
  • change the import string for the operator into: from airflow.providers.apache.sqoop.operators.sqoop import SqoopOperator

Here is the full list of the providers available:

https://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html

And this is the provider you are looking for:

https://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html#apache-airflow-providers-apache-sqoop

Upvotes: 2

Related Questions