CODEWITHSUNDEEP
airflowgoogle-cloud-composer
conradlee
conradlee

Reputation: 13705

How do I set AWS credentials using the Google Cloud Composer CLI?

I want to access AWS services from my cloud-composer-managed airflow system. I don't want to set this up from the airflow UI -- I want to do it using the Google Cloud SDK which serves as the CLI.

Is my only option here to use the CLI's env-variables option? If so, is it sufficient for me to just set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env varibles? Or do I need to set lots of airflow-specific connection variables, e.g. for s3 AIRFLOW_CONN_S3_DEFAULT?

Upvotes: 2

Views: 1796

Answers (1)

Feng Lu
Feng Lu

Reputation: 691

You can tunnel airflow commands through Cloud Composer SDK CLI using gcloud composer environment run. For example:

gcloud config set composer/location {your-env-location}
gcloud composer environments run {your-env-name} connections -- -a --conn_id your-aws-id --conn_type aws {any other connection details}

So if you wanted to update an the aws_default connection of an environment called my-env, you'd first delete the pre-existing entry (this connection_id is pre-populated by airflow) with the following command:

gcloud composer environments run my-env connections -- -d --conn_id aws_default

and then add your credentials as follows:

gcloud composer environments run data-lake connections -- -a --conn_id aws_default --conn_type aws --conn_extra '{"region_name": "us-east-1", "aws_access_key_id":"YOUR_ACCESS_KEY_HERE", "aws_secret_access_key": "YOUR_SECRET_KEY_HERE"}'

Of course, you'll want to update the AWS region name and the creds accordingly.

Upvotes: 5

Related Questions