CODEWITHSUNDEEP
pythonairflow
Shubam sachdeva
Shubam sachdeva

Reputation: 77

Airflow UI not able to find the provider modules

I have installed Airflow on the server which is running Ubuntu and python 3.8. I'm trying to import a simple dag in Airflow UI to list the files in the bucket.

from airflow import DAG
from airflow.providers.amazon.aws.operators.s3_copy_object import S3CopyObjectOperator
from airflow.providers.amazon.aws.operators.s3_list import S3ListOperator
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator

from datetime import datetime

default_args = {
    'start_date':datetime(2020,1,1)
}


with DAG('S3_to_S3', schedule_interval='@daily',default_args=default_args,catchup=False) as dag:
    list_files = S3ListOperator(
        task_id = 'list_S3_bucket',
        aws_conn_id='S3_Connection',
        bucket='xxxxx'
    )

list_files

But it fails to import in Airflow UI and throws the exception:

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/airflow/airflow/dags/S3_to_S3.py", line 4, in <module>
    from airflow.providers.amazon.aws.operators.s3_list import S3ListOperator
ModuleNotFoundError: No module named 'airflow.providers.amazon'

I have already installed the amazon provider package and its dependencies using pip, but Airflow UI fails to find those packages. To verify if provider is installed I imported the amazon package in python console and it was successfully imported.

Python 3.8.9
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
>>>
>>> from airflow.providers.amazon.aws.operators.s3_list import S3ListOperator
>>> from airflow.providers.amazon.aws.operators.s3_copy_object import S3CopyObjectOperator
>>> import airflow.providers.amazon
>>>

Do I need to change any configuration in airflow or set any environment variable?

Upvotes: 3

Views: 7179

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 15961

As discussed in comments the issue happens because the provider is installed in different path than Airflow resulting in Airflow not finding the provider library:

/usr/local/lib/python3.8/dist-packages

.local/lib/python3.8/site-packages/

The solution is to clean up the environment and install the provider in the same path of Airflow. This has been also discussed in Github issue

Upvotes: 2

Related Questions