Kubernetes - internal load balancing safety

Question

We're expanding our microservice herd application and i was looking into Kubernetes for our needs. Before my dive into the modern orchestration i was thinking about service discovery in the following way:

• Cluster is bootstrapped with some kind of distributed service registry (Consul in our case)
• Every service is launched with service registry endpoints passed in somehow
• Every service self-registers itself in registry
• Whenever service needs some other service addresses, it fetches contact points from registry

In that case, if any service fails or some kind of network disruption occurs, client service may proceed with next contact point and eventually succeed (in case it is not totally cut off). As far as i've understood, kubernetes uses completely different model:

• All pods are self-registered in kubernetes
• Kubernetes provides single load balancer instance to pass traffic through to services
• Load balancer itself may be discovered via environment variables or DNS query (and that may result in creepy things such as fetching port from DNS records or just stale environment variable)

And that confuses me a little. If i'm correct (feel free to tell me i'm not if that's the case), this basically turns load balancer into SPOF that may stop whole application the moment it dies. Am i right? Are there any guarantees made by Kubernetes that such situation won't happen or would be resolved in <N> <time units>?

Answer 1

The in-cluster load balancer in Kubernetes (kube-proxy) is distributed among all of your cluster's nodes. It syncs all service endpoints to iptables rules on the node. Kube-proxy is healthchecked by kubelet, and will be restarted if it is unhealthy for 30 seconds (that's configurable, I think). The actual traffic does not go through the kube-proxy binary, so if kube-proxy does get stuck, the worst thing that happens is your view of service endpoints gets stale. Take a look at the docs and the Virtual IPs section of the Service Docs, Services FAQs, and slides 46-61 of this kubernetes networking deck for more details on kube-proxy.

For service discovery, each service is accessible by name through kube-dns. Pods have kube-dns in their search path, so no environment variable magic is necessary. Slides 68-83 of that same deck has a full walkthrough of DNS->Virtual IP->Pod traffic.

So, the load balancer is not really a SPOF. It should mostly share fate with the workloads running on the same node. Kube-dns could be a SPOF for service discovery, but it can be replicated however much you like.