CODEWITHSUNDEEP
google-kubernetes-enginegoogle-cloud-composer
Twist
Twist

Reputation: 542

How to setup Google Cloud Composer to be able to launch pods on a GKE Autopilot cluster

I would like to be able to use a Google Cloud Composer cluster to launch kubernetes pods from its DAGs onto a separate GKE Autopilot cluster instead of onto the GKE cluster of Cloud Composer.

I have created a GKE autopilot cluster with "control plane global access" set to disabled and only allowing certain authorised networks to connect to the control plane. (based on the recommended security best practices in the documentation)

My pods all fail to launch with the following error message:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='3X.XXX.XXX.XX6', port=443): Max retries exceeded with url: /api/v1/namespaces/sink/pods?labelSelector=dag_id%3Dtest_dag%2Cexecution_date%3D2021-03-17T212059.4745700000-f0b251c80%2Ctask_id%3Dtest_sync (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0f3f6a4e80>: Failed to establish a new connection: [Errno 110] Connection timed out',)

I am using the GKEStartPodOperator which previously was able to start pods on a GKE cluster that was self managed (not autopilot) and which did not have "control plane global access" disabled.

Is there any documentation about how to setup Composer to be able to connect to a GKE autopilot cluster that is not exposing global access to the control plane and launch pods?

Upvotes: 1

Views: 1748

Answers (1)

hexacyanide
hexacyanide

Reputation: 91789

Even with GKE Autopilot, you can use the same set of operators that was originally written for use with normal GKE clusters, such as GKEStartPodOperator. Since the error you are seeing is a timeout to the Kubernetes control plane, it is most likely that your authorized networks setting does not include the addresses used by your Cloud Composer environment's workers.

If you are using a standard Composer environment (non-private IP), you will need to ensure that GCP ranges are included within your authorized networks (because your environment's nodes are assigned ephemeral, public addresses).

If you are using a private IP environment, then you can use private connectivity to reach the Kubernetes control plane, or alternatively, you can configure a Cloud NAT to allow your environment to reach network resources using a static IP address. In the latter case, the IP address of the NAT would need to be included within your authorized networks settings.

Upvotes: 0

Related Questions