CODEWITHSUNDEEP
amazon-web-serviceskubernetesairflowmwaa
Tommy
Tommy

Reputation: 13672

AWS MWAA; How to install specific version of AWS CLI V2

So, we can see from this fun issue: https://github.com/aws/aws-cli/issues/4947

that AWS CLI 2 is never going to be pushed to pip.

In AWS MWAA (Managed AIrflow), we are not given the ability to specify apt-get level installs. We get a requirements.txt, only, and AWS V2 is not available via that. Meaning, we have no interface or medium to do what the Amazon representative in that issue talks about, which is to use the package manager (from Amazon's MWAA).

Recently, we have hit issues with authenticating to our kubernetes clusters, because the AWS CLI in various versions hardcodes the auth version: https://github.com/aws/aws-cli/blob/1.20.11/awscli/customizations/eks/get_token.py#L77

In other versions, it's hardcoded to alpha etc. Which means depending on what aws cli version a worker comes up with, it may or may not match what your kubeconfig in S3 for MWAA has. We've been hitting transient mismatches like

{{refresh_config.py:71}} ERROR - exec: plugin api version client.authentication.k8s.io/v1alpha1 does not match client.authentication.k8s.io/v1beta1

and, if we "flip" our kubeconfig,

{{refresh_config.py:71}} ERROR - exec: plugin api version client.authentication.k8s.io/v1beta1 does not match client.authentication.k8s.io/v1alpha1

Since we use the kubernetes executor from Airflow, we need get-token:

  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - thor
      command: aws

but this get-token returns a different version of the auth sometimes as discussed.

Currently, we have pinned awscli hard in requirements.txt to 1.20.11. However, AWS Says to stop using V1: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-mac.html

So, with all of this said, how can I install a specific version of AWS CLI 2 on my AWS MWAA workers? Help from Amazon would be appreciated; I will file an AWS Support ticket there too.

Upvotes: 4

Views: 937

Answers (2)

Gabe
Gabe

Reputation: 6085

You'll want to look at Shell Launch (startup) scripts to install external tools using bash and Yum for example.

sudo yum -y install libaio

See links below

Upvotes: 0

ferruzzi
ferruzzi

Reputation: 141

MWAA doesn't currently support installing a specific AWS CLI version. This is a limitation that the MWAA team is aware of.

Upvotes: 2

Related Questions