Istio Proxy unable to connect to Istio Pilot on EKS

Question

I have installed bookinfo on EKS according to the instructions here and here.

While verifying that the application was installed correctly, i received 000 when trying to bring up the product page. After checking my network connections VPC/Subnets/Routing/SecurityGroups, I have narrorwed the issue down to being an istio networking issue.

Upon further investigation, I logged into the istio-sidecar container for productpage and have noticed the following error.

[2019-01-21 09:06:01.039][10][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:41] Unable to establish new stream
[2019-01-21 09:06:28.150][10][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, no healthy upstream

This led me to to notice that the istio-proxy is pointing to the istio-pilot.istio-system:15007 address for discovery. Only the strange thing was, the kubernetes istio-pilot.istio-system service does not seem to be exposing port 15007 as shown below.

[procyclinsur@localhost Downloads]$ kubectl get svc istio-pilot --namespace=istio-system
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                 AGE
istio-pilot   ClusterIP   172.20.185.72   <none>        15010/TCP,15011/TCP,8080/TCP,9093/TCP   1d

Infact none of the services from the istio-system namespace seem to expose that port. I am assuming that this istio-pilot.istio-system address is the one used for gRPC and would like to know how to fix this as it seems to be pointing to the wrong address; please correct me if I am wrong.

Relevant Logs

istio-proxy

2019-01-21T09:04:58.949152Z info    Version root@6f6ea1061f2b-docker.io/istio-1.0.5-c1707e45e71c75d74bf3a5dec8c7086f32f32fad-Clean
2019-01-21T09:04:58.949283Z info    Proxy role: model.Proxy{ClusterID:"", Type:"sidecar", IPAddress:"10.20.228.89", ID:"productpage-v1-54b8b9f55-jpz8g.default", Domain:"default.svc.cluster.local", Metadata:map[string]string(nil)}
2019-01-21T09:04:58.949971Z info    Effective config: binaryPath: /usr/local/bin/envoy
configPath: /etc/istio/proxy
connectTimeout: 10s
discoveryAddress: istio-pilot.istio-system:15007
discoveryRefreshDelay: 1s
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: productpage
zipkinAddress: zipkin.istio-system:9411

Answer 1

I wanted to post the solution to my issue.

Problem:

EKS DNS was not properly working which is why none of the other solutions (while very good!!) worked for me.

Cause:

When an AWS VPC is first created VPC DNS settings are not properly setup for EKS. The following VPC settings are required to be enabled by EKS.

• DNS resolution: Enabled
• DNS hostnames: Disabled <-- Default VPC Settings

Solution:

Set DNS hostnames to Enabled and DNS begins to work as expected.

Answer 2

Ignore the gRPC warnings they are not meaningful. Make sure you did the kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

If you do kubectl exec $(kubectl get pod --selector app=ratings --output jsonpath='{.items[0].metadata.name}') -c istio-proxy -- ps -ef you will see an entry like --discoveryAddress istio-pilot.istio-system:15011. That is the address the sidecar uses to contact Pilot and SHOULD match an entry you see using kubectl -n istio-system get service istio-pilot.

If the discoveryAddress matches a Pilot port you can test networking. You can't easily curl on the discovery address but if you do kubectl exec $(kubectl get pod --selector app=ratings --output jsonpath='{.items[0].metadata.name}') -c istio-proxy -- curl https://istio-pilot.istio-system:15011 and you get a timeout then there is a communication problem.

The discovery address comes from Istio configuration. If you do kubectl -n istio-system get cm istio-sidecar-injector and the age is older than your Istio install there may have been a problem with upgrading an older Istio version.

Answer 3

To verify that Istio works correctly, send a request to the productpage from another pod, e.g. from ratings:

kubectl exec -it $(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}') -c ratings -- curl productpage:9080/productpage | grep -o "<title>.*</title>"

The output should be: <title>Simple Bookstore App</title>

If you get the correct output, the problem is probably with your Ingress definitions.

Verify carefully that you followed the steps specified here and here.

Answer 4

Sending you the link to the Istio docs which may be really helpful when debugging Istio.