How does gRPC connection work on kubernetes service ClusterIP

Question

I'm probably missing something in my understanding of networking and gRPC. I want to start with an example to explain how I understand the networking.

I have a deployment with 3 replicas named app in default namespace they have pod IP:

10.3.0.1, 10.3.0.2, 10.3.0.3

I have a ClusterIP service for the deployment called app-service, it has an IP of:

10.24.0.0

The kube-dns will have a record that maps the app-service.default.svc.cluster.local -> 10.24.0.0. The kube-proxy on every node will see the configuration and update the netfilter with mapping 10.24.0.0 -> 10.3.0.1, 10.3.0.2, 10.3.0.3.

Now in my cluster I have another client pod that makes a gRPC call to app-service.default.svc.cluster.local.

What I expect to happen is app-service.default.svc.cluster.local will resolve to a single IP 10.24.0.0, and gRPC will create a single sub channel and try to establish a long-live connection.

This call will get out of the pod and to the node and go pass the netfilter at which point 10.24.0.0 becomes 10.3.0.1 and finally arrives on the first pod.

Now there is a second call from the client, it goes through the netfilter at which point 10.24.0.0 becomes 10.3.0.2, this call ends up on another pod which had no idea they had a connection established?

Also I see these blogs that mention gRPC will create sticky session to a single pod IP, but I thought the pod IP will not be resolved inside the application/grpc client but in the netfilter.

Answer 1

I think you could find some answers from the Load Balancing in gRPC.

Load-balancing policies fit into the gRPC client workflow in between name resolution and the connection to the server.

• On startup, the gRPC client issues a name resolution request for the server name. The name will resolve to a list of IP addresses, a service config that indicates which client-side load-balancing policy to use (e.g., round_robin or grpclb) and provides a configuration for that policy and a set of attributes (channel args in C-core).
• The client instantiates the load balancing policy and passes its configuration from the service config, the list of IP addresses, and the attributes.
• The load balancing policy creates a set of subchannels for the IP addresses of the servers (which might be different from the IP addresses returned by the resolver; see below). It also watches the subchannels' connectivity states and decides when each subchannel should attempt to connect.
• For each RPC sent, the load balancing policy decides which subchannel (i.e., which server) the RPC should be sent to.

When the IP of pods is changed due to pods restart, now gRPC will try to re-resolve whenever a subconn goes into transient failure. Here is one related discussion and code

Answer 2

While your thinking process might sound correct, it works differently.

When you have a single clusterIP service it has a single A DNS record. So connection will look like this:

So at the given moment there will be only one/single persistent connection, other two pods won't be used. kube-proxy has this mapping inside using netfilter, but it's not a proper physical loadbalancer, no PODs' IPs will be exposed and client will be aware only about service's clusterIP. If a new connection is created, already created connection will be used with a help of grpc multiplexing.

What you need here is a headless service type. What it does is:

For headless Services that define selectors, the endpoints controller creates Endpoints records in the API, and modifies the DNS configuration to return A records (IP addresses) that point directly to the Pods backing the Service.

And schema will look differently now:

In other words, with headless service type service's FQDN will be resolved in 3 PODs IP addresses where gRPC client can establish different connections.

For more details please read a good article (where I took these pictures from) - Balancing gRPC Traffic in K8S Without a Service Mesh.

Also service mesh can be used for loadbalancing (proxy loadbalancing),