GKE Ingress backends marked as "Unhealthy"

Question

I am learning how to use an ingress to expose my application on Google Kubernetes Engine. I followed several tutorials and had a rough setup of what is needed. However, I have no clue why are my service is marked as unhealthy despite them being accessible from the NodePort service I defined directly.

Here is my deployment file: (I removed some data but the most of it remains the same)

--
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  name: "deployment-1"
  namespace: "default"
    spec:
      containers:
      - name: myContainer
        image: "myImage/"
        readinessProbe:
          httpGet:
            path: /app1
            port: 8080
          initialDelaySeconds: 70      
        livenessProbe:
          httpGet:
            path: /app1
            port: 8080
          initialDelaySeconds: 70    
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - mountPath: opt/folder/libs/jdbc/
          name: lib
      volumes:
      - name: lib
        persistentVolumeClaim:
          claimName: lib                            
---

As I read, I need a ReadinessProbe and LivinessProbe in order for GKE to run a health check on a path I defined, and by using my own defined path shown as /app1 here (which will return a 200 OK), the generated health check should pass. I set an initial delay of 70s as buffer time for the tomcat server running in the image to startup.

Next I created a NodePort service as the backend for the Ingress: I tested by connecting to the node's public IP and nodeport of this service, and it successfully runs.

---
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  ports:
  - name: my-port
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: deployment-1
  type: NodePort

and then the Ingress manifest file:

Here I have reserved a static IP address with the name "gke-my-static-ip" as well as created a managedCertificate "gke-my-certificate" with a domain name "mydomain.web.com". This has also been configured on the DNS records to point it to that reserved static IP.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gke-my-ingress-1
  annotations:
    kubernetes.io/ingress.global-static-ip-name: gke-my-static-ip
    networking.gke.io/managed-certificates: gke-my-certificate
spec:
  rules:
  - host: mydomain.web.com
    http:
      paths:
      - backend:
          serviceName: my-service
          servicePort: my-port

The ingress creates 2 backends by default, one on the /healthz path and one with my custom defined path /app1. The healthz path returns 200 OK, but my custom defined path is failing to connect. I have checked on the firewall rules and have allowed ports tcp30000-32767.

Checking on stackdriver, the health check tries to access my LoadBalancer's IP with the /app1 path but it seems to always return a 502 error.

Am I missing any steps in my setup?

Attached ingress,endpoints:

NAME                                        HOSTS                    ADDRESS          PORTS   AGE
ingress.extensions/gke-my-ingress-1   mydomain.web.com   <IP_ADDRESS>   80      3d15h

NAME                         ENDPOINTS           AGE
endpoints/kubernetes          <IP_ADDRESS>443   9d
endpoints/presales-service    <IP_ADDRESS>:8080     4d16h

kubectl get ingress:

Name:             gke-my-ingress-1
Namespace:        default
Address:          <IP_ADDRESS>
Default backend:  default-http-backend:80 (<IP_ADDRESS>)
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  mydomain.web.com
                          /   my-service:my-port (<IP_ADDRESS>:8080)
Annotations:
  ingress.kubernetes.io/target-proxy:                k8s-tp-default-gke-my-ingress-1--d8d0fcf4484c1dfd
  ingress.kubernetes.io/url-map:                     k8s-um-default-gke-my-ingress-1--d8d0fcf4484c1dfd
  kubernetes.io/ingress.global-static-ip-name:       gke-my-static-ip
  networking.gke.io/managed-certificates:            gke-my-certificate
  ingress.gcp.kubernetes.io/pre-shared-cert:         mcrt-e7dd5612-e6b4-42ca-91c9-7d9a86abcfb2
  ingress.kubernetes.io/forwarding-rule:             k8s-fw-default-gke-my-ingress-1--d8d0fcf4484c1dfd
  ingress.kubernetes.io/ssl-cert:                    mcrt-e7dd5612-e6b4-42ca-91c9-7d9a86abcfb2
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.global-static-ip-name":"gke-my-static-ip","networking.gke.io/managed-certificates":"gke-my-certificate"},"name":"gke-my-ingress-1","namespace":"default"},"spec":{"rules":[{"host":"mydomain.web.com","http":{"paths":[{"backend":{"serviceName":"my-service","servicePort":"my-port"},"path":"/"}]}}]}}

  ingress.kubernetes.io/backends:               {"k8s-be-30242--d8d0fcf4484c1dfd":"HEALTHY","k8s-be-30310--d8d0fcf4484c1dfd":"UNHEALTHY"}
  ingress.kubernetes.io/https-forwarding-rule:  k8s-fws-default-gke-my-ingress-1--d8d0fcf4484c1dfd
  ingress.kubernetes.io/https-target-proxy:     k8s-tps-default-gke-my-ingress-1--d8d0fcf4484c1dfd

Answer 1

By tinkering with the Readiness and Liveliness probe by adding a successThreshold and FailureThreshold, I managed to get my ingress working. It might be because my application needs a little more buffer time to run.

        readinessProbe:
          httpGet:
            path: /app1/
            port: 8080
          periodSeconds: 5
          timeoutSeconds: 60 
          successThreshold: 1
          failureThreshold: 3
          initialDelaySeconds: 70      
        livenessProbe:
          httpGet:
            path: /app1/
            port: 8080
          periodSeconds: 5
          timeoutSeconds: 60
          successThreshold: 1
          failureThreshold: 3          
          initialDelaySeconds: 70