CODEWITHSUNDEEP
pythongoogle-cloud-platformairflowgcloudgoogle-cloud-composer
rodrigo-silveira
rodrigo-silveira

Reputation: 13086

How to update gcloud on Google Cloud Composer worker nodes?

There's a similar question here but from 2018 were the solution requires changing the base image for the workers. Another suggestion is to ssh into each node and apt-get install there. This doesn't seem useful because when auto scale spawns new nodes, you'd need to do it again and again.

Anyway, is there a reasonable way to upgrade the base gcloud in late 2020?

Upvotes: 0

Views: 971

Answers (1)

hexacyanide
hexacyanide

Reputation: 91799

Because task instances run in a shared execution environment, it is generally not recommended to use the gcloud CLI within Composer Airflow tasks, when possible, to avoid state or version conflicts. For example, if you have multiple users using the same Cloud Composer environment, and either of them changes the active credentials used by gcloud, then they can unknowingly break the other's workflows.

Instead, consider using the Cloud SDK Python libraries to do what you need to do programmatically, or use the airflow.providers.google.cloud operators, which may already have what you need.

If you really need to use the gcloud CLI and don't share the environment, then you can use a BashOperator with a install/upgrade script to create a prerequisite for any tasks that need to use the CLI. Alternatively, you can build a custom Docker image with gcloud installed, and use GKEPodOperator or KubernetesPodOperator to run a Kubernetes pod to run the CLI command. That would be slower, but more reliable than verifying dependencies each time.

Upvotes: 3

Related Questions