CODEWITHSUNDEEP
pythonamazon-web-servicesairflowmwaa
Armando Fa Menor
Armando Fa Menor

Reputation: 1

EcsCreateClusterOperator import error in MWAA (Apache Airflow)

I am trying to use class EcsCreateClusterOperator in MWAA but I get the following error:

Broken DAG: [/usr/local/airflow/dags/etl_basic_ecs/etl_ecs_basic_dag.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/airflow/dags/etl_basic_ecs/etl_ecs_basic_dag.py", line 5, in <module>
    from airflow.providers.amazon.aws.operators.ecs import (
ImportError: cannot import name 'EcsCreateClusterOperator' from 'airflow.providers.amazon.aws.operators.ecs' (/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/ecs.py)

This library works perfectly in my local environment installed with:

pip install apache-airflow-providers-amazon==5.0.0

I also added this library (version 5.0.0) to my requirements.txt file before deploying to MWAA, in the UI seems like the provider is installed:

Screenshot of installed Amazon Provider 5.0.0

The log of the update shows that the library is downloaded:

Installing library log

Here is the documentation of the class: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/operators/ecs/index.html this class was introduced recently (version 5.0.0).

By default MWAA (Airflow 2.2.2) has the Amazon Provider with an older version (2.4), seems like Airflow is really loading the older version of the library.

Any help would be appreciated, thank you.

Upvotes: 0

Views: 1019

Answers (2)

Manuel Israel De Aguinaga
Manuel Israel De Aguinaga

Reputation: 1

I had the same issue with EksCreateClusterOperator:

from airflow.providers.amazon.aws.operators.eks import ( ImportError: cannot import name 'EksCreateClusterOperator' from 'airflow.providers.amazon.aws.operators.eks' (/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/eks.py)

Looks like is case sensitive. So instead of using EcsCreateClusterOperator change it to ECSCreateClusterOperator (capital ECS) how it is showing on airflow github. That fixed my issue.

Upvotes: 0

Andrew Nguonly
Andrew Nguonly

Reputation: 2621

MWAA with Airflow version 2.2.2 is constrained to apache-airflow-providers-amazon==2.4.0. See the constraints file for more details.

References:

  1. Airflow Versions (MWAA)
  2. Constraints file

Upvotes: 0

Related Questions