CODEWITHSUNDEEP
nginxkubernetesgrpckubernetes-ingressgrpc-java
RamPrakash
RamPrakash

Reputation: 3234

Kubernetes - Ingress with gRPC

I have a very simple grpc-server. When I expose them directly via NodePort everything works just fine. I do not have any issues with this.

grpc-client --> NodePort --> grpc-server

I changed the NodePort service to ClusterIP service and tried to use ingress controller to route the traffic to the grpc-server. The set up is like this. ingress and grpc-server are part of k8s cluster. grpc-client is outside - which is my local machine.

grpc-client --> ingress --> clusterip --> grpc-server

My ingress is like this.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web-ingress
  namespace: test
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;    
spec:
  rules:
  - http:
      paths:
      - backend:
         serviceName: grpc-server
         servicePort: 6565

Whenever I send a request, I get en exception like this in the client side.

io.grpc.StatusRuntimeException: INTERNAL: HTTP/2 error code: FRAME_SIZE_ERROR
Received Rst Stream

Ingress log is like this

127.0.0.1 - - [28/Jul/2020:03:52:38 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - c9d46af55aa8a2b450f9f72e9d550023
127.0.0.1 - - [28/Jul/2020:03:52:38 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - a9e0f02851473467fc553f2a4645377d
127.0.0.1 - - [28/Jul/2020:03:52:40 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - fcc0668f300bd5e7e12325ae7c26fa9d
127.0.0.1 - - [28/Jul/2020:03:52:41 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - dea2b8eb8f6884ac512654af3c2e6b47
127.0.0.1 - - [28/Jul/2020:03:52:43 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 0357acafce928ee05aaef1334eed1b69
127.0.0.1 - - [28/Jul/2020:03:52:44 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - df9b75d79d9177e39a901dfa020912f2
127.0.0.1 - - [28/Jul/2020:03:52:44 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - bb27acd5b86fd057af1ca8fe93f7d857
127.0.0.1 - - [28/Jul/2020:03:52:44 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 2ced9207652fc564bb94e0d4a14ae6fc
127.0.0.1 - - [28/Jul/2020:03:52:45 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - b99fab13df2f64a6a0a901237dadfa27
127.0.0.1 - - [28/Jul/2020:03:52:45 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.002 [] [] - - - - 40cb75f4e4b3df28be66044a0825f1c6
127.0.0.1 - - [28/Jul/2020:03:52:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 89cac6709278b4e54b0349075500bf74
127.0.0.1 - - [28/Jul/2020:03:52:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - 0b032199a053069df185c620de497c90
127.0.0.1 - - [28/Jul/2020:03:52:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 5a07b1e24ea3e8e2066a7ed8950f7040
127.0.0.1 - - [28/Jul/2020:03:53:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - 0f63eec0d0ccb9cc87c2fbb1fdfae48f

Looks like the grpc-server never received any request as I do not see any log there.

What is going on?

Upvotes: 3

Views: 7383

Answers (1)

acid_fuji
acid_fuji

Reputation: 6853

That error you encountered comes from the default configuration of the nginx ingress controller (this is taken from the ingress github config):

// Enables or disables the HTTP/2 support in secure connections
// http://nginx.org/en/docs/http/ngx_http_v2_module.html
// Default: true
UseHTTP2 bool `json:"use-http2,omitempty"`

This can be also verified by checking your ingress controller settings :

kubectl exec -it -n <name_space> <nginx_ingress_controller_pod> -- cat /etc/nginx/nginx.conf 

## start server _
   server {
          server_name _ ;

          listen 80 default_server reuseport backlog=511 ;
          listen 443 default_server reuseport backlog=511 ssl http2 ;

You're trying to connect through port 80 for which there is no http2 support in default configuration. GRCP works only with http2 so you should try to make a request using port 443.

Let me know if it works.

Upvotes: 4

Related Questions