GKE Cluster with hybrid private/public subnetwork

Question

We currently have a public GKE cluster on which many data processing tasks are deployed. These tasks use a lot of bandwith because they download/upload very large files. For some of our tasks, they need to have static IP for 3rd parties connection. In a public GKE cluster, the outbound IP will be nodes IP, nodes are deleted, recreated with different IPs. We found that we can use a private cluster with NAT to have a static outbound IP. The problem is the pricing ($0.045 per GB). The only solution we found is to have one public cluster and one private cluster, the first one for tasks that doesn't need static IP and the second one for tasks that need one.

I there a way to have public node pools and private node pools in the same cluster ? Or any way to use the NAT only when we need it ?

Thank you so much, If you have any question, please ask ! Have a good evening.

Answer 1

There are two options to a "hybrid" public/private GKE cluster. Both involve configuring a "public" cluster.

• Use an IP masquerade Agent to prevent some nodes from directly reaching the internet:
  • https://rajathithanrajasekar.medium.com/google-cloud-public-gke-clusters-egress-traffic-via-cloud-nat-for-ip-whitelisting-7fdc5656284a
  • https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent
• Use a hybrid cluster, i.e., containing private and public node-pools
  • This option is significantly simpler, and probably the recommended approach here
  • To create a "private" node-pool inside a "public" cluster, that will egress through the NAT, add the following config:

    network_config {
      enable_private_nodes = true
    }
    

  • https://chimbu.medium.com/private-node-pools-in-standard-public-gke-clusters-9190ff62bef
  • https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_node_pool#enable_private_nodes

Answer 2

You cannot mix and match public and private node pools in GKE.

One solution would be to run your own NAT:

• Deploy a proxy (HA, NGINX...) on a set of GCV VM's. Ideally in different zones for redundancy
• Deploy an Internal LoadBalancer in front of these VM's
• Assign Static IP's to those VM's (when you reserve a Public IP it's your's).
• Use those VM's to proxy outbound calls to your 3rd party service via the Internal Loadbalancer. If your outbound calls are HTTP(S), this is simple as exporting env variable HTTP_PROXY and setting it to your LB:Port