gRPC Load Balancing

Question

I have read the Load Balancing page at https://github.com/grpc/grpc/blob/master/doc/load-balancing.md to start of, but still am confused as to the right approach to loadbalancing between back end GRPC instances. We are deploying multiple gRPC 'microservice' instances and want our other gRPC clients to be able to be routed between them. We are deploying these as pods in kubernetes (actually Google Container Engine).

Can anyone explain the 'recommended' approach to loadbalancing gRPC client requests between the gRPC servers? It seems that clients need to be aware of the endpoints - is it not possible to take advantage of the inbuilt LoadBalancer in Container Engine to help?

Answer 1

At Fulcrum we had a similar case.

We also built a product completely on microservices architecture and decided to use gRPC framework. We then immediately faced the problem of how to load balance traffic using the Application Load Balancer (Layer 7). As it turned out – AWS does not support HTTP 2.0 yet.

Why we used Envoy

As the commenter above pointed out, Envoy is a great option here. It's been developed at Lyft and it's great. It's very lightweight compared to the alternatives and it's easy to incorporate into a microservices architecture.

Basically, we used the standard Frontend Envoy load balancer. Besides, there're also load balancers called sidecars. Each of them is placed just near each instance of service. They work specifically with the service nearby & they are also connected with each other.

As a result, Frontend proxy always has a connection or it can create one when needed. It uses a 'round-robin' model for sending requests.

Here's a complete description of our experience with load balancing gRPC > https://fulcrum.rocks/blog/load-balancer-grpc-aws We also created a very simple service that you can download from our GitHub and launch the playground on your own machine: https://github.com/fulcrum-rocks/grpc-load-balancing-playground

Here's how we configured Frontend Proxy

static_resources:
  listeners:
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: $LISTEN_PORT
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                codec_type: http2
                stat_prefix: ingress_grpc
                http2_protocol_options: {}
                route_config:
                  name: service_route
                  virtual_hosts:
                    - name: service_host
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/"
                            grpc: {}
                          route:
                            cluster: service_cluster
                            timeout:
                              seconds: 0
                              nanos: 0
                http_filters:
                  - name: envoy.router
                    config: {}
  clusters:
    - name: service_cluster
      connect_timeout: 0.25s
      type: strict_dns
      lb_policy: round_robin
      http2_protocol_options: {}
      hosts:
        - socket_address:
            address: $SERVICE_DISCOVERY_ADDRESS
            port_value: $SERVICE_DISCOVERY_PORT
admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 80

Answer 2

As mentioned by Samarendra

You can check Envoy,Originally built at Lyft.

It has first class support for HTTP/2 and gRPC for both incoming and outgoing connections.

• L7 load balancing for gRPC
• cluster discovery service (ex. consul)

We have been using envoy for our gRPC services at gojek and it is serving us well under quite high load.

Answer 3

If in AWS/ECS a https://aws.amazon.com/blogs/aws/new-network-load-balancer-effortless-scaling-to-millions-of-requests-per-second/ will work to round robin connections between your target groups.

You do need to do something to listen to the healthcheck which NLBs can only do in HTTP1. Also you get to terminate your own SSL.

Answer 4

For Load balancing between grpc server, kubernates default load balancing wont help as it is a L4 load balancer. you would be requiring L7 load balancer.

Why L7?

grpc uses http2 where connections are persistent and request will be sent through same connection. L4 load balancer will load balance using tcp connections, but u need a load balance at request level so we would require a L7 load balancer. especially when communication is between grpc servers.

there are couple of options, you could use Linkered/Envoy for this, they are work good with kubernates and provide a good service mesh also.

to expose your services to outside work you can use nghttpx and nghttpx Ingress controller.

you can also use client side load balancing, but i don't see a good merit in that.

Answer 5

I can't talk about kubernetes, but regarding gRPC loadbalancing, there are basically two approaches:

1. For simple usecases, you can enable round robin over the list of addresses returned for a given name (ie, the list of IPs returned for service.foo.com). The way to do this is language-dependent. For C++, you'd use grpc::ChannelArguments::SetLoadBalancingPolicyName with "round_robin" as the argument (in the future it'd also be possible to select via "service configuration", but the design for how to encode that config in DNS records hasn't been finalized yet).
2. Use the grpclb protocol. This is suitable for more complex deployements. This feature required the c-ares DNS resolver, which #11237 introduces (this PR is very close to being merged). This is the piece that's missing for making grpclb work in open source. In particular:
   • Have a look at this document. It goes over the DNS configuration changes needed to control which addresses are marked as balancers. It's currently a "proposal", to be promoted to a doc shortly. It can be taken quite authoritatively, it's what #11237 is implementing for balancer discovery.
   • Write a regular gRPC server (in any language) implementing the load balancer protocol. This is the server to be marked in your DNS records as a balancer (as described in the aforementioned document), with which the client's grpclb will talk to to obtain the list of backend addresses (what's called server_lists). It's up to you to make the logic inside this balancer as simple or as complex as you want.
   • The client would use the DNS name of the balancer when creating a channel. Note also that your balancer DNS name may point to several addresses. If one or more of them are marked as balancers, grpclb will be used. Which balancer will be picked up if there's more than one? The first one the client connects to.

Let me know if you have any questions.

Answer 6

You might want to check out nghttpx and the nghttpx Ingress controller projects. These projects make it possible to perform L7 load balancing for HTTP/2 streams.

Also have a look at https://cloud.google.com/endpoints/docs/architecture-overview, for some information about the endpoint management bundled in Google Cloud Platform.