gRPC behind Istio Gateway

Question

I'm having trouble debugging why my gRPC servers cannot be reached behind an Istio gateway. Here's how to reproduce:

# Use enough resources for Istio.
minikube start --memory=16384 --cpus=4
# Set up Istio in ambient mode.
istioctl install --set profile=ambient --skip-confirmation
# Set up Gateway API.
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

Now, apply the following objects, using a pre-built example gRPC server that listens on port 50051:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: foo-gateway
spec:
  gatewayClassName: istio
  listeners:
    - name: foo-grpc
      protocol: HTTP
      port: 50051
      allowedRoutes:
        kinds:
          - kind: GRPCRoute
---
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: foo-api-v1
spec:
  parentRefs:
    - name: foo-gateway
  rules:
    - backendRefs:
        - name: foo-api-v1
          port: 50051
---
apiVersion: v1
kind: Service
metadata:
  name: foo-api-v1
  labels:
    app: foo-api-v1
spec:
  ports:
    - name: http
      port: 50051
  selector:
    app: foo-api-v1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: foo-api-v1
  labels:
    account: foo-api-v1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo-api-v1
  labels:
    app: foo-api-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: foo-api-v1
  template:
    metadata:
      labels:
        app: foo-api-v1
    spec:
      serviceAccountName: foo-api-v1
      # Use a stock example gRPC server.
      containers:
        - name: foo-api-v1
          image: 'docker.io/grpc/java-example-hostname'
          imagePullPolicy: Always
          ports:
            - containerPort: 50051

Once it's all up and running, we can invoke RPC's directly on foo-api-v1 service like so:

kubectl port-forward svc/foo-api-v1 8080:50051
grpcurl -plaintext -format text -d 'name: "Jimbo"' localhost:8080 helloworld.Greeter/SayHello
# message: "Hello Jimbo, from foo-api-v1-5678866d86-sp9wt"

It works, cool. However, I can't reach it through the gateway:

kubectl port-forward svc/foo-gateway-istio 9090:50051
grpcurl -plaintext -format text -d 'name: "Jimbo"' localhost:9090 helloworld.Greeter/SayHello
# Error invoking method "helloworld.Greeter/SayHello": failed to query for service descriptor "helloworld.Greeter": server does not support the reflection API

It's not just an issue with reflection. It still does not work even if you specify the service definition with grpcurl:

grpcurl -plaintext -format text -d 'name: "Jimbo"' -proto helloworld.proto localhost:9090 helloworld.Greeter/SayHello
# ERROR:
#   Code: Unavailable
#   Message: upstream connect error or disconnect/reset before headers. reset reason: protocol error

I've tried inspecting the traffic with Kubeshark and found a gRPC request from foo-gateway-istio-77bc48898f-nrg9w/envoy to itself. It has the following POST data (text/plain):

{"field_1":"Jimbo"}

This is odd. It seems like Istio's gateway is doing automatic JSON transcoding and using a default field name ("field_1") instead of a proper field name. That would explain "protocol error" from upstream. But I'm not sure what to do about this.