CODEWITHSUNDEEP
grpctraefiktraefik-ingress
jmaitrehenry
jmaitrehenry

Reputation: 2420

Traefik as an ingress controller + Server GRPC - UNKNOWN: No status received

I use traefik as an ingress controller in AKS, I have a grpc service that run correctly locally, but I have some problem behind traefik.

When the GRPC server return an error, I receive it correctly, but when it send a normal response, I didn’t receive it:

[email protected]:443> client.Ping({}, metadata, pr)
EventEmitter {}
[email protected]:443> 
Error:  { Error: 2 UNKNOWN: No status received
    at Object.exports.createStatusError (/usr/local/lib/node_modules/grpcc/node_modules/grpc/src/common.js:91:15)
    at Object.onReceiveStatus (/usr/local/lib/node_modules/grpcc/node_modules/grpc/src/client_interceptors.js:1204:28)
    at InterceptingListener._callNext (/usr/local/lib/node_modules/grpcc/node_modules/grpc/src/client_interceptors.js:568:42)
    at InterceptingListener.onReceiveStatus (/usr/local/lib/node_modules/grpcc/node_modules/grpc/src/client_interceptors.js:618:8)
    at callback (/usr/local/lib/node_modules/grpcc/node_modules/grpc/src/client_interceptors.js:845:24) code: 2, metadata: {}, details: 'No status received' }

vs

grpcServer@localhost:10000> client.Ping({}, metadata, pr)
EventEmitter {}
grpcServer@localhost:10000> 
{
  "response": "PONG"
}

Error:

[email protected]:443> client.Ping({}, pr)
EventEmitter {}
[email protected]:443> 
Error:  { Error: 16 UNAUTHENTICATED: incorrect serial number
    at Object.exports.createStatusError (/usr/local/lib/node_modules/grpcc/node_modules/grpc/src/common.js:91:15)
    at Object.onReceiveStatus (/usr/local/lib/node_modules/grpcc/node_modules/grpc/src/client_interceptors.js:1204:28)
    at InterceptingListener._callNext (/usr/local/lib/node_modules/grpcc/node_modules/grpc/src/client_interceptors.js:568:42)
    at InterceptingListener.onReceiveStatus (/usr/local/lib/node_modules/grpcc/node_modules/grpc/src/client_interceptors.js:618:8)
    at callback (/usr/local/lib/node_modules/grpcc/node_modules/grpc/src/client_interceptors.js:845:24)
  code: 16,
  metadata:
   { 'content-length': '0', date: 'Wed, 06 Feb 2019 21:32:38 GMT' },
  details: 'incorrect serial number' }

The k8s service yml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: velcloud-grpc
  namespace: production
  annotations:
    kubernetes.io/ingress.class: traefik
    ingress.kubernetes.io/protocol: h2c
    traefik.protocol: h2c
spec:
  rules:
    - host: grpc.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: velcloud-grpc
              servicePort: grpc

Edit: More information

After some debugging, I inspected the received response:

{ client_close: true,
  metadata: { date: [ 'Wed, 06 Feb 2019 23:48:06 GMT' ] },
  read: <Buffer 0a 04 50 4f 4e 47>,
  status:
   { code: 2,
     details: 'No status received',
     metadata: Metadata { _internal_repr: {} } } } { code: 2,
  details: 'No status received'
}

The only problem here is the status, the read content the right response once deserialized: { response: 'PONG' }

I don't know why the status is set to 2 (UNKNOWN) and not 0 (OK).

Upvotes: 0

Views: 1821

Answers (1)

jmaitrehenry
jmaitrehenry

Reputation: 2420

In my Traefik configuration, I use the retry middleware([retry]) and Traefik v1.7.8 have a bug:

When the retry middleware has already sent the headers, use the original headers map to be able to send trailers.

https://github.com/containous/traefik/pull/4442

I manually build the v1.7 branch with this fix and everything works again! Thanks a lot to Containous (Damien and Julien!) for helping to debug this issue!

Upvotes: 2

Related Questions