Ingress Nginx rules doesn't get followed always

Question

I got 2 spring app pods deployed in k8 cluster. 1 replica each. Both have their cluster-ip services exposing the services.

1. /upstream/agentLabs/makeotherTierCall should forward to prevlab deployment /agentLabs/makeOtherTierCall
2. /downstream/basics/hello should forward to newlab deployment /basics/hello

I am using ingress-nginx to redirect the traffic with following rules.

ingress-service.yml

apiVersion: networking.k8s.io/v1
# UPDATE API
kind: Ingress
metadata:
  namespace: javaspace
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: 'nginx'
    nginx.ingress.kubernetes.io/use-regex: 'true'
    # ADD ANNOTATION
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    # UPDATE ANNOTATION
spec:
  rules:
    - http:
        paths:
          - path: /upstream?(.*)
            # UPDATE PATH
            pathType: Prefix
            # ADD PATHTYPE
            backend:
              service:
                # UPDATE SERVICE FIELDS
                name: prevlab-cluster-ip-service
                port:
                  number: 8080
          - path: /downstream?(.*)
            # UPDATE PATH
            pathType: Prefix
            # ADD PATHTYPE
            backend:
              service:
                # UPDATE SERVICE FIELDS
                name: newlab-cluster-ip-service
                port:
                  number: 8080

The issue is when I make any call (both 1 and 2) sometimes the ingress controller forwards the traffic correctly and sometimes it doesn't and I get 404. Basically, what I am observing is that alternatively the traffic is getting routed to both cluster-ip services one after the other.

for eg. if I make /upstream/agentLabs/makeotherTierCall request 2 times, at first it forwards wrongly to newlab cluster-ip service (giving back 404), and then next it forwards correctly to prevlab cluster-ip service (giving back 200).

ingress-nginx controller setup

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.1/deploy/static/provider/cloud/deploy.yaml

ingress-controller logs

192.168.65.3 - - [24/Sep/2021:10:04:21 +0000] "GET /upstream/agentLabs/makeOtherTierCall HTTP/2.0" 200 12 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 45 0.008 [javaspace-prevlab-cluster-ip-service-8080] [] 10.1.0.181:8080 12 0.007 200 24ae53531ce4d7109004f81e79534ca4
192.168.65.3 - - [24/Sep/2021:10:04:21 +0000] "GET /upstream/agentLabs/makeOtherTierCall HTTP/2.0" 404 286 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 45 0.006 [javaspace-prevlab-cluster-ip-service-8080] [] 10.1.0.179:8080 286 0.005 404 1a32efb3a9dc21cce01b908afeb0248a
192.168.65.3 - - [24/Sep/2021:10:04:22 +0000] "GET /upstream/agentLabs/makeOtherTierCall HTTP/2.0" 200 12 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 45 0.007 [javaspace-prevlab-cluster-ip-service-8080] [] 10.1.0.181:8080 12 0.007 200 542bd4abba25be5a416deca1152cb29b
192.168.65.3 - - [24/Sep/2021:10:04:22 +0000] "GET /upstream/agentLabs/makeOtherTierCall HTTP/2.0" 404 286 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 45 0.008 [javaspace-prevlab-cluster-ip-service-8080] [] 10.1.0.179:8080 286 0.007 404 aaea8e3bf60dab81ef7454d51863a22d 

192.168.65.3 - - [24/Sep/2021:10:07:23 +0000] "GET /downstream/basics/hello HTTP/2.0" 404 306 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 35 0.009 [javaspace-newlab-cluster-ip-service-8080] [] 10.1.0.181:8080 306 0.009 404 178bd79cc71b73f7a337a2652322d65f
192.168.65.3 - - [24/Sep/2021:10:07:23 +0000] "GET /downstream/basics/hello HTTP/2.0" 200 12 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 35 0.002 [javaspace-newlab-cluster-ip-service-8080] [] 10.1.0.179:8080 12 0.003 200 280d2b7974ae38a467237cc6cf437b98
192.168.65.3 - - [24/Sep/2021:10:07:24 +0000] "GET /downstream/basics/hello HTTP/2.0" 404 306 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 35 0.008 [javaspace-newlab-cluster-ip-service-8080] [] 10.1.0.181:8080 306 0.008 404 4e237232b4c53c9bb90e7bb59c15e916
192.168.65.3 - - [24/Sep/2021:10:07:24 +0000] "GET /downstream/basics/hello HTTP/2.0" 200 12 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 35 0.003 [javaspace-newlab-cluster-ip-service-8080] [] 10.1.0.179:8080 12 0.002 200 be0cf532a972ec0d783e15869470f579

Using Docker Desktop's Kubernetes. Kubernetes 1.21.2

What am I doing wrong here? Is this default behavior? Do I need to do some additional config?

Answer 1

Nevermind. I had done a very small but very stupid mistake. In case someone comes wandering around here: I had used the same label for my 2 backend services. And for each of the cluster-IP services used the same selector. Hence, I was seeing inconsistent behavior.