CODEWITHSUNDEEP
google-cloud-platform
Jan
Jan

Reputation: 43

GCE/GKE NAT gateway route kills ssh connection

im trying to setUp a NAT Gateway for Kubernetes Nodes on the GKE/GCE. I followed the instructions on the Tutorial (https://cloud.google.com/vpc/docs/special-configurations chapter: "Configure an instance as a NAT gateway") and also tried the tutorial with terraform (https://github.com/GoogleCloudPlatform/terraform-google-nat-gateway)

But at both Tutorials (even on new created google-projects) i get the same two errors:

As soon as i tag the gke-node-instances, so that the configured route applies to them, the SSH connection is no longer possible.

Upvotes: 2

Views: 997

Answers (2)

Dan Cowell
Dan Cowell

Reputation: 385

You've already found the solution to the first problem: tag the nodes with the correct tag, or manually create a route targeting the instance group that is managing your GKE nodes.

Regarding the SSH issue:

This is answered under "Caveats" in the README for the NAT Gateway for GKE example in the terraform tutorial repo you linked (reproduced here to comply with StackOverflow rules).

The web console mentioned below uses the same ssh mechanism as kubectl exec internally. The short version is that as of time of posting it's not possible to both route all egress traffic through a NAT gateway and use kubectl exec to interact with pods running on a cluster.

Update @ 2018-09-25:

There is a workaround available if you only need to route specific traffic through the NAT gateway, for example, if you have a third party whose service requires whitelisting your IP address in their firewall.

Note that this workaround requires strong alerting and monitoring on your part as things will break if your vendor's public IP changes.

If you specify a strict destination IP range when creating your Route in GCP then only traffic bound for those addresses will be routed through the NAT Gateway. In our case we have several routes defined in our VPC network routing table, one for each of our vendor's public IP addresses.

In this case the various kubectl commands including exec and logs will continue to work as expected.

A potential workaround is to use the command in the snippet below to connect to a node and use docker exec on the node to enter a container. This of course means you will need to first locate the node your pod is running on before jumping through the gateway onto the node and running docker exec.

Caveats

The web console SSH will no longer work, you have to jump through the NAT gateway machine to SSH into a GKE node:

eval ssh-agent $SHELL
ssh-add ~/.ssh/google_compute_engine
CLUSTER_NAME=dev
REGION=us-central1
gcloud compute ssh $(gcloud compute instances list --filter=name~nat-gateway-${REGION} --uri) --ssh-flag="-A" -- ssh $(gcloud compute instances list --filter=name~gke-${CLUSTER_NAME}- --limit=1 --format='value(name)') -o StrictHostKeyChecking=no

Source: https://github.com/GoogleCloudPlatform/terraform-google-nat-gateway/tree/master/examples/gke-nat-gateway

Upvotes: 2

avivl
avivl

Reputation: 401

You can use kubeip in order to assign IP addresses https://blog.doit-intl.com/kubeip-automatically-assign-external-static-ips-to-your-gke-nodes-for-easier-whitelisting-without-2068eb9c14cd

Upvotes: 0

Related Questions