GRPC Client not able to find Virtual hostname when deployed with istio DestinationRule and VirtualService

Question

I have an Grpc application ( java + maven). Both grpc server and client are deployed with istio on minikube cluster. When grpc client tries to make a grpc call to server it gets Error

SEVERE: Servlet.service() for servlet [ServerSubsetServlet] in context with path [/grpc_client] threw exception
io.grpc.StatusRuntimeException: UNAVAILABLE: Failed to find virtual host matching hostname: grpc-server-service.grpc-istio.svc.cluster.local:9090
    at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
    at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
    at com.example.grpcproto.GrpcIstioRoutingServerGrpc$GrpcIstioRoutingServerBlockingStub.getServerSubset(GrpcIstioRoutingServerGrpc.java:157)
    at org.example.GrpcClientHandler.getServerSubset(GrpcClientHandler.java:39)
    at org.example.ServerSubsetTestServlet.doGet(ServerSubsetTestServlet.java:23)
    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564)
    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)

Server Side Destination Rule and Virtual Service

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: grpc-server-destination
  namespace: grpc-istio
spec:
  host: grpc-server-service.grpc-istio.svc.cluster.local
  subsets:
  - name: server-1
    labels:
      server: server-1
  - name: server-2
    labels:
      server: server-2
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grpc-server-route
  namespace: grpc-istio
spec:
  hosts:
    - grpc-server-service.grpc-istio.svc.cluster.local
  http:
  - match:
    - headers:
        SERVER:
          exact: "server-1"
    route:
    - destination:
        host: grpc-server-service.grpc-istio.svc.cluster.local
        subset: server-1
  - match:
    - headers:
        SERVER:
          exact: "server-2"
    route:
    - destination:
        host: grpc-server-service.grpc-istio.svc.cluster.local
        subset: server-2

Client side Deployment yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grpc-client
  namespace: grpc-istio
  labels:
    name: grpc-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grpc-client
  template:
    metadata:
      labels:
        app: grpc-client
      annotations:
        sidecar.istio.io/inject: 'true'
        proxy.istio.io/config: 'holdApplicationUntilProxyStarts: true'
        inject.istio.io/templates: 'grpc-agent'
    spec:
      containers:
        - name: grpc-client
          image: grpc-client:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: GRPC_SERVER_ADDRESS
              value: "xds:///grpc-server-service.grpc-istio.svc.cluster.local:9090"
          ports:
            - containerPort: 8080

Without envoy proxy side car the GRPC client and server are able to communicate using channel, but when sidecar container is deployed and virtual service along with destination rule is used for request routing the client get this above error

Unable to get what is wrong with Virtual service / DestinationRule!!

Expected Behaviour:

Setup is like we can hit grpc client which also exposes the 8080 port for HTTP calls , when we hit the http call with a param server-1 or server-2 it directs the grpc call to server with the same corresponding label

There are 2 deployments for server with different labels each deployment has 2 replicas.

So when grp client make the request with server-1 in header metadata the call should go to the pod with server: server-1 label, and similarly when request is made using server-2 header the call should go to pods with label of server: server-2

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grpc-server-1
  namespace: grpc-istio
  labels:
    name: grpc-server-1
    server: server-1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: grpc-server
  template:
    metadata:
      labels:
        app: grpc-server
        server: server-1
      annotations:
        sidecar.istio.io/inject: 'true'
        proxy.istio.io/config: 'holdApplicationUntilProxyStarts: true'
        inject.istio.io/templates: 'grpc-agent'
    spec:
      containers:
        - name: grpc-server
          image: grpc-server:latest
          imagePullPolicy: Never
          env:
            - name: SERVER
              value: "SERVER_1"
          ports:
            - containerPort: 9090
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grpc-server-2
  namespace: grpc-istio
  labels:
    name: grpc-server-2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: grpc-server
  template:
    metadata:
      labels:
        app: grpc-server
        server: server-2
      annotations:
        sidecar.istio.io/inject: 'true'
        proxy.istio.io/config: 'holdApplicationUntilProxyStarts: true'
        inject.istio.io/templates: 'grpc-agent'
    spec:
      containers:
        - name: grpc-server
          image: grpc-server:latest
          imagePullPolicy: Never
          env:
            - name: SERVER
              value: "SERVER_2"
          ports:
            - containerPort: 9090

Answer 1

In your deployment spec on the client side I see something called GRPC_SERVER_ADDRESS being set to xds:///grpc-server-service.grpc-istio.svc.cluster.local:9090 which will only work in the proxyless mode. In the proxy mode (with Envoy sidecar) you need to remove the xds:/// prefix so that the access goes through DNS lookup first and then the sidecar which will use xDS lookup.