CODEWITHSUNDEEP
kubernetesgoogle-kubernetes-engineistioistio-gatewaygoogle-cloud-armor
nomadic_squirrel
nomadic_squirrel

Reputation: 644

Backends are not healthy in GKE cluster with Istio and Cloud Armor

I'm trying to play with Cloud Armor Adaptive protection in front of a GKE cluster. I found this guide from contino-engineering and this one from always up always on. However, both of those use a simple hello world example and I'm trying to deploy a (hopefully?) more realistic deployment by using the bookinfo example from istio. (I have similar results using the onlineboutique as well.)

The issue is that the backends never get healthy. I'm assuming it is because the health checkers can't reach the health check service, but I'm new to all this and unsure how to verify.

This is my TF for deploying the cluster. Note that the ip_allocation_policy is equivalent to the --enable-ip-aliases gcloud parameter... I think. I'm not enabling CloudRun here in the hopes of simplifying things.

cluster tf

resource "google_container_cluster" "primary" {
  provider = google-beta
  name     = "my-gke-cluster"
  location = "us-central1"

  # We can't create a cluster with no node pool defined, but we want to only use
  # separately managed node pools. So we create the smallest possible default
  # node pool and immediately delete it.
  remove_default_node_pool = true
  initial_node_count       = 4

  networking_mode = "VPC_NATIVE"
  ip_allocation_policy {
    // Set to blank to have a range chosen with the default size
    cluster_ipv4_cidr_block = ""
  }

  addons_config {
    istio_config {
      disabled = false
      auth = "AUTH_MUTUAL_TLS"
    }
  }
}

After getting creds, I apply my backendconfig to allow(?)/enable(?) the health check.

cat <<EOF > /tmp/backendconfig.yaml
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  healthCheck:
    requestPath: /healthz/ready
    port: 15021
    type: HTTP
  securityPolicy:
    name: my-security-policy
EOF

kubectl apply -n istio-system -f /tmp/backendconfig.yaml

Then I patch the ingressgateway:


cat <<EOF > /tmp/patch.yaml
spec:
   type: NodePort
metadata:
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
    cloud.google.com/backend-config: '{"default": "my-backendconfig"}'
status: null
EOF
kubectl patch svc istio-ingressgateway -n istio-system --patch-file /tmp/patch.yaml

Finally, apply the ingress resource

cat <<EOF > /tmp/istio-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
    - http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: istio-ingressgateway
                port: 
                  number: 80
EOF

kubectl apply -n istio-system -f /tmp/istio-ingress.yaml

Rollout new pods for good measure:

kubectl rollout restart deploy -n istio-system

After waiting 10+ minutes, I see the new load balancer in the Google console showing the unhealthy backend: Load balancer

Upvotes: 1

Views: 749

Answers (1)

nomadic_squirrel
nomadic_squirrel

Reputation: 644

@Anant Swaraj I tracked this down to a mismatched port for my VirtualService:

# ---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: health
spec:
  hosts:
  - "*"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - headers:
        user-agent:
          prefix: GoogleHC
      method:
        exact: GET
      uri:
        exact: /
    rewrite:
      authority: istio-ingressgateway:15020
      uri: /healthz/ready
    route:
      - destination:
          host: istio-ingressgateway
          port:
            number: 15020

I used 15020 there, but 15021 in my BackendConfig. Not really sure where I got either after looking at so many posts so many times.

But as @Anant Swaraj mentions, there will need to be a firewall rule in addition.

Upvotes: 0

Related Questions