CODEWITHSUNDEEP
airflowgcloudgoogle-cloud-composergcloud-cli
Adam Hopkinson
Adam Hopkinson

Reputation: 28793

Setting composer (airflow) bucket using gcloud CLI

I've been creating a composer environment using the console interface, and then setting three variables using the cli (using examples from the QuickStart doc):

gcloud composer environments run etl --location europe-west1 variables -- --set gcp_project projectname
gcloud composer environments run etl --location europe-west1 variables -- --set gcs_bucket gs://bucketname
gcloud composer environments run etl --location europe-west1 variables -- --set gce_zone europe-west1-b

This doesn't set the bucket for some reason, but all three variables appear under Admin > Variables

I was hoping to combine all this into one step, and set the composer bucket too - so built the following CLI command:

gcloud composer environments create etl --location=europe-west1 --env-variables=gcp_project=projectname,gcs_bucket=gs://bucketname,gce_zone=europe-west1-b --service-account=serviceemail --zone=europe-west1-b

But this doesn't set the bucket and doesn't even create the variables.

How do I kick off a composer environment from the CLI with the bucket name and variables set?

Upvotes: 1

Views: 567

Answers (1)

Feng Lu
Feng Lu

Reputation: 691

Composer doesn't support user-provided GCS bucket, each newly created environment comes with its own GCS bucket.

Upvotes: 1

Related Questions