Installing kiali on GKE gives backend NotFound error

Question

I've installed kiali operator and tried to load the UI from the URL(x.x.x.x/kiali) on Ingress. Following is the text I'm getting when loading the url.

response 404 (backend NotFound), service rules for [ /kiali/ ] non-existent

All my cluster components are green as follows. Any idea ?

Answer 1

I encountered the same problem and finally got it working. After some comparison tests and lots of confusion, here's what's going on and how to solve it:

1. The latest official istio 1.7.4 tutorial gives a sample ingress yaml that contains a new attribute pathType which is only available on GKE 1.18 and later:

kind: Ingress
...
  - host: my-kiali.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          serviceName: kiali
          ...

Now, GKE 1.18 is only available in GCP's RAPID release channel. For the REGULAR channel which is the default in GCP console, the highest is GKE 1.17 right now. If so, that is obvious to notice and fix because 1.17's kubectl apply will complain about syntax error. What is not obvious is that, in 1.17 and earlier, we'd need to append a wild card to path so that it reads:

      - path: /*

Without that wild card *, ingress will return the error message you mentioned to the browser (the HTTP reply status code for this request is indeed 404 as suggested in the text body):

response 404 (backend NotFound), service rules for [ /kiali/ ] non-existent 

That's the easy part and answers your question (hopefully). Now onto the next problem although it's not really part of your question...

2. Suppose we got beyond problem 1. We now deploy the ingress with kubectl apply .... We then need to wait out a few minutes for GCP to create the External Load Balancer. Various internet literatures say to wait until the external IP address is assigned -- you can check that on GCP console for the load balancer or:

$ kubectl describe ingress istio-system -n istio-system | grep Address

But I don't think that's quite enough -- and this phenomenon isn't restricted to Istio. After the IP address is assigned, you still need to further wait until GCP reports that all backend health checks are in HEALTHY status. You can check it either in CLI:

$ kubectl get ingress istio-system -n istio-system -o yaml | grep ingress.kubernetes.io/backends

    ingress.kubernetes.io/backends: '{...,"k8s1-898cbc37-istio-system-kiali-20001-dfe8fc73":"HEALTHY",...}'

or in GCP console:

How long is that extra wait? For me, from the time the IP is assigned to the time the backend is HEALTHY, it took 7 minutes:

$ kubectl describe ingress istio-system -n istio-system
...
Events:
  Type    Reason  Age    From                     Message
  ----    ------  ----   ----                     -------
  Normal  ADD     ...    loadbalancer-controller  istio-system/istio-system
  Normal  CREATE  7m21s  loadbalancer-controller  ip: 34.120.142.25

That is despite the readinessProbe being much shorter:

$ cat istio-1.7.4/samples/addons/kiali.yaml
        ...
        readinessProbe:
          httpGet:
            ...
          initialDelaySeconds: 5
          periodSeconds: 30

The big difference between actual wait and periodSeconds can cause quite a bit of confusion during operation.

When a browser tries to hit http://my-kiali.io (BTW, must add that hostname to C:\Windows\System32\drivers\etc\hosts), here's what the browser will experience over different phases on the wait:

• During most of that 7-minute wait (after IP is assigned), the browser will get HTTP 404. Note that this isn't the same error as in your question. My 404 here is the HTTP status for the very first URL http://my-kiali.io/, meaning that the backend isn't in HEALTHY status -- the HTTP request never reached the Kiali pod. In contrast, your 404 is actually preceded by an HTTP 302 (redirect from / to /kiali/) meaning that the backend is HEALTHY (that's how the HTTP 302 reply/redirect was generated -- by the Kiali pod) but the load balancer can't find a forwarding rule for the redirected URL path /kiali/ (because the rule lacked a wild card *).

• Near the end of that 7-minute wait, for a brief time, the browser might get HTTP 502 ("Bad Gateway"). This is simply when the Load Balancer is transitioning.

• After the 7-minute wait, the browser will first get HTTP 302 ("Redirect") which redirects from / to /kiali/ and followed by HTTP 200 (after successful redirect). And the Kiali GUI console shows up successfully on the browser.

Problem solved!

P.S.: If you're indeed following Istio official tutorial, you might encounter 2 other (unrelated) errors:

a) timeoutSeconds vs. periodSeconds:

Error during sync: error running backend syncing routine: googleapi: Error 400: Invalid value for field 'resource.timeoutSec': '30'. TimeoutSec should be less than checkIntervalSec., invalid

The solution is to edit all istio-1.7.4/samples/addons/*.yaml so that timeoutSeconds is less than periodSeconds.

b) Ingress servicePort for my-istio-tracing.io. The sample kind: Ingress shown in the tutorial has the wrong port number. The correct port number should be 80:

  - host: my-istio-tracing.io
    http:
      paths:
      - ...
        backend:
          ...
          servicePort: 9411  # <<=== This should be 80.