Istio on GKE in Autopilot mode

Question

Hi there I was reviewing the GKE autopilot mode and noticed that in cluster configureation istio is disabled and I'm not able to change it. Also installation via istioctl install fail with following error

 error   installer       failed to update resource with server-side apply for obj MutatingWebhookConfiguration//istio-sidecar-injector: mutatingwebhookconfigurations.admissionregistration.k8s.io "istio-sidecar-injector" is forbidden: User "something@example" cannot patch resource "mutatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope: GKEAutopilot authz: cluster scoped resource "mutatingwebhookconfigurations/" is managed and access is denied

Am I correct or it's not possible to run istio in GKE autopilot mode?

Answer 1

According to the documentation this should be possible with GKE 1.21:

In GKE version 1.21.3-gke.900 and later, you can create validating and mutating dynamic admission webhooks. However, Autopilot modifies the admission webhooks objects to add a namespace selector which excludes the resources in managed namespaces (currently, kube-system) from being intercepted. Additionally, webhooks which specify one or more of following resources (and any of their sub-resources) in the rules, will be rejected:

• group: "" resource: nodes
• group: certificates.k8s.io resource: certificatesigningrequests
• group: authentication.k8s.io resource: tokenreviews

https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#webhooks_limitations

Answer 2

TL;DR

It is not possible at this moment to run istio in GKE autopilot mode.

Conclusion

If you are using Autopilot, you don't need to manage your nodes. You don't have to worry about operations such as updating, scaling or changing the operating system. However, the autopilot has a number of limitations.

Even if you are trying to install istio with a command istioctl install, istio will not be installed. You will then see the following message:

This will install the Istio profile into the cluster. Proceed? (y/N) y

✔ Istio core installed
✔ Istiod installed
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition Deployment/istio-system/istio-ingressgateway

• Pruning removed resources 2021-05-07T08:24:40.974253Z warn installer retrieving resources to prune type admissionregistration.k8s.io/v1beta1, Kind=MutatingWebhookConfiguration: mutatingwebhookconfigurations.admissionregistration.k8s.io is forbidden: User "something@example" cannot list resource "mutatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope: GKEAutopilot authz: cluster scoped resource "mutatingwebhookconfigurations/" is managed and access is denied not found Error: failed to install manifests: errors occurred during operation

This command failed, bacuse for sidecar injection, installer tries to create a MutatingWebhookConfiguration called istio-sidecar-injector. This limitation is mentioned here.

For more information you can also read this page.

Answer 3

It is not possible to create mutating admission webhooks according to documentation

You cannot create custom mutating admission webhooks for Autopilot clusters

Since Istio uses mutating webhooks to inject its sidecars, it will probably not work and it is also consistent with the error you get.