CODEWITHSUNDEEP
nginxkubernetesgrpckubernetes-ingressnginx-ingress
bhnat
bhnat

Reputation: 107

kubernetes ingress with gRPC and HTTP

I have an application deployed to kubernetes (AKS) where I have a mix of gRPC and http services. I initially added the route for a new gRPC service to the existing ingress which was previously serving only http. That didn't work and digging into it, I read that we need to add the nginx.ingress.kubernetes.io/backend-protocol: GRPC annotation, and that it applied to all routes, so we would need two separate ingress. I'm currently getting an exception io.grpc.internal.ManagedChannelImpl$NameResolverListener error trying to connect to the gRPC service with message nodename nor servname provided, or not known. I'm guessing that though when multiple paths within an Ingress match a request, precedence is given first to the longest matching path, that doesn't apply across the both ingress. So I would need to either use different hosts, or change the /* path so that it didn't also match /results? Or is there something else that I need to change in my configuration?

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - {{ .Values.ingress.hosts.host }}
    secretName: {{ .Values.ingress.tls.secretName }}
  rules:
  - host: {{ .Values.ingress.hosts.host }}
    http:
      paths:
      - path: /graphql
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
            port:
              number: 80
      - path: /graphql/*
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
            port:
              number: 80
      - path: /*
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-grpc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - {{ .Values.ingress.hosts.host }}
    secretName: {{ .Values.ingress.tls.secretName }}
  rules:
  - host: {{ .Values.ingress.hosts.host }}
    http:
      paths:
      - path: /results
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.externalResults.host }}
            port:
              number: 9000

Upvotes: 1

Views: 5322

Answers (1)

bhnat
bhnat

Reputation: 107

This wound up being resolved by creating a second host name that pointed to our k8s cluster. I changed the route for the grpc service to be the root path and pathType of ImplementationSpecific.

      - path: /
        pathType: ImplementationSpecific

Both host names needed to be included in the tls section of both ingress. I was getting an SSL exception after changing the route but not updating the hosts in the tls section of each ingress.

Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
    at io.grpc.Status.asRuntimeException(Status.java:533)
    at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
  | => cat io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:721)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
stderr: 
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshakeException(ReferenceCountedOpenSslEngine.java:1771)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:776)
    at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:511)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1079)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:970)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1443)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
stderr: 
    at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
stderr: 
    at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ... 1 more
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching grpc.aks.dev.app.cycleautomation.com found.
stderr: 
    at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214)
    at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
    at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:462)
    at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:428)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:261)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
    at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSslTlsv13X509ExtendedTrustManager.checkServerTrusted(OpenSslTlsv13X509ExtendedTrustManager.java:223)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:261)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:700)
    at io.grpc.netty.shaded.io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:595)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1202)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1324)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1367)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1380)
    ... 21 more
    Suppressed: javax.net.ssl.SSLHandshakeException: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1287)
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1248)
        ... 25 more

The final yaml looked like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - {{ .Values.ingress.hosts.host }}
    - {{ .Values.ingress.grpc.host }}
    secretName: {{ .Values.ingress.tls.secretName }}
  rules:
  - host: {{ .Values.ingress.hosts.host }}
    http:
      paths:
      - path: /graphql
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
            port:
              number: 80
      - path: /graphql/*
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
            port:
              number: 80
      - path: /*
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-grpc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - {{ .Values.ingress.hosts.host }}
    - {{ .Values.ingress.grpc.host }}
    secretName: {{ .Values.ingress.tls.secretName }}
  rules:
  - host: {{ .Values.ingress.hosts.host }}
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.externalResults.host }}
            port:
              number: 9000

Then I was able to connect to the grpc service over port 443 w/ tls enabled and just using the host name with no path in my connection.

Upvotes: 2

Related Questions