GKE Ingress defaultBackend not working despite healthy pods and correct NEG configuration

Question

I have a GKE Ingress configured with a defaultBackend to handle requests for undefined hosts. The pods are healthy, but the default backend does not respond. Here’s my setup:

---

Configuration Details

1. Deployment for default-backend:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: default-backend
     namespace: ydt
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: default-backend
     template:
       metadata:
         labels:
           app: default-backend
       spec:
         containers:
         - name: nginx
           image: nginx:stable
           ports:
             - containerPort: 80
           livenessProbe:
             httpGet:
               path: /
               port: 80
             initialDelaySeconds: 5
             periodSeconds: 5
           readinessProbe:
             httpGet:
               path: /
               port: 80
             initialDelaySeconds: 5
             periodSeconds: 5
   

   • Pods are healthy:

     kubectl get pods -n ydt -l app=default-backend
     NAME                              READY   STATUS    RESTARTS   AGE
     default-backend-7c58d6c88-abcde   1/1     Running   0          2h
     

2. Service for default-backend:

   apiVersion: v1
   kind: Service
   metadata:
     name: default-backend-service
     namespace: ydt
     annotations:
       cloud.google.com/neg: '{"ingress": true}'  # NEGs enabled
   spec:
     type: ClusterIP
     ports:
       - port: 80
         targetPort: 80
     selector:
       app: default-backend
   

3. Ingress Configuration:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ydt-ingress
     namespace: ydt
     annotations:
       kubernetes.io/ingress.class: "gce"
       networking.gke.io/managed-certificates: "certificate"
   spec:
     defaultBackend:
       service:
         name: default-backend-service
         port:
           number: 80
     rules:
       # Other host rules (omitted for brevity)
   

---

What I’ve Checked

• Pods are Running and probes succeed (kubectl logs shows no errors).
• Service selector matches the Deployment’s labels.
• NEGs are created in Google Cloud Console (Network Services > Network Endpoint Groups).
• Firewall rules allow traffic from 130.211.0.0/22 and 35.191.0.0/16 (GCP health check ranges).

---

The Problem

In the following image you can see the backend service regarding the default backend:

There is no healthy pod running.

There is a google cloud example to create a custom Default Backend service and I don't see any diference with mine

Question:
Why is the defaultBackend not working, even though pods are healthy and NEGs are enabled? Are there hidden configurations in GKE/GCP that I’m missing?

---

Answer 1

You can try these troubleshooting steps for default backend not working in GKE Ingress setup:

Since your configuration seems fine. You might have to check the logs of Ingress controller to confirm that the request is received by the controller and rerouting the traffic to the default backend.

You can get the logs from the GKE Ingress controller with:

$kubectl logs -n kube-system -l app=gke-ingress

You need to check the health check logs as well, follow this document to enable the health check logging and check the state of your health check by using this document as reference.

Check this official gcp health check troubleshooting document to troubleshoot your health check

Note: Sometimes there may be issues with the GKE ingress controller not properly syncing with the Google Cloud load balancer or health checks. Try deleting and recreating the Ingress resource and the associated services.

Delete the Ingress:

kubectl delete ingress ydt-ingress -n ydt

Reapply the Ingress configuration:

kubectl apply -f your-ingress-config.yaml