Minikube Nginx Ingress not finding service endpoint

Question

I'm having some trouble getting the Nginx ingress controller working in my Minikube cluster. It's likely to be some faults in Ingress configuration but I cannot pick it out.

First, I deployed a service and it worked well without ingress.

kind: Service
apiVersion: v1
metadata:
  name: online
  labels:
    app: online
spec:
  selector:
    app: online
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 5001
  type: LoadBalancer

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: online
  labels:
    app: online
spec:
  replicas: 1
  selector:
    matchLabels:
      app: online
  template:
    metadata:
      labels:
        app: online
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "online"
        dapr.io/app-port: "5001"
        dapr.io/log-level: "debug"
        dapr.io/sidecar-liveness-probe-threshold: "300"
        dapr.io/sidecar-readiness-probe-threshold: "300"
    spec:
      containers:
      - name: online
        image: online:latest
        ports:
        - containerPort: 5001
        env:
        - name: ADDRESS
          value: ":5001"
        - name: DAPR_HTTP_PORT
          value: "8080"
        imagePullPolicy: Never

Then check its url

minikube service online --url
http://192.168.49.2:32323

It looks ok for requests.

curl http://192.168.49.2:32323/userOnline
OK

After that I tried to apply nginx ingress offered by minikube. I installed ingress and run an example by referring to this and it's all ok.

Lastly, I configured my Ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: online-ingress
  annotations:
spec:
  rules:
    - host: online
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: online
                port:
                  number: 8080

And changed /etc/hosts by adding line

192.168.49.2    online

And Test:

curl online/userOnline
502 Bad Gateway

The logs are like this:

192.168.49.1 - - [26/Aug/2021:09:45:56 +0000] "GET /userOnline HTTP/1.1" 502 150 "-" "curl/7.68.0" 80 0.002 [default-online-8080] [] 172.17.0.5:5001, 172.17.0.5:5001, 172.17.0.5:5001 0, 0, 0 0.004, 0.000, 0.000 502, 502, 502 578ea1b1471ac973a2ac45ec4c35d927
2021/08/26 09:45:56 [error] 2514#2514: *426717 upstream prematurely closed connection while reading response header from upstream, client: 192.168.49.1, server: online, request: "GET /userOnline HTTP/1.1", upstream: "http://172.17.0.5:5001/userOnline", host: "online"
2021/08/26 09:45:56 [error] 2514#2514: *426717 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.49.1, server: online, request: "GET /userOnline HTTP/1.1", upstream: "http://172.17.0.5:5001/userOnline", host: "online"
2021/08/26 09:45:56 [error] 2514#2514: *426717 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.49.1, server: online, request: "GET /userOnline HTTP/1.1", upstream: "http://172.17.0.5:5001/userOnline", host: "online"
W0826 09:45:56.918446       7 controller.go:977] Service "default/online" does not have any active Endpoint.
I0826 09:46:21.345177       7 status.go:281] "updating Ingress status" namespace="default" ingress="online-ingress" currentValue=[] newValue=[{IP:192.168.49.2 Hostname: Ports:[]}]
I0826 09:46:21.349078       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"online-ingress", UID:"b69e2976-09e9-4cfc-a8e8-7acb51799d6d", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"23100", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

I found the error is very about annotations of Ingress. If I changed it to:

  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1

The error would be:

404 page not found

and logs:

I0826 09:59:21.342251       7 status.go:281] "updating Ingress status" namespace="default" ingress="online-ingress" currentValue=[] newValue=[{IP:192.168.49.2 Hostname: Ports:[]}]
I0826 09:59:21.347860       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"online-ingress", UID:"8ba6fe97-315d-4f00-82a6-17132095fab4", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"23760", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
192.168.49.1 - - [26/Aug/2021:09:59:32 +0000] "GET /userOnline HTTP/1.1" 404 19 "-" "curl/7.68.0" 80 0.002 [default-online-8080] [] 172.17.0.5:5001 19 0.000 404 856ddd3224bbe2bde9d7144b857168e0

Other infos.

NAME     TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
online   LoadBalancer   10.111.34.87        8080:32323/TCP   6h54m

The example I mentioned above is a NodePort service and mine is a LoadBalancer, that's the biggest difference. But I don't know why it does not work for me.

moonkotte · Accepted Answer

Moving this out of comments so it will be visible.

Ingress

Main issue was with path in ingress rule since application serves traffic on online/userOnline. If requests go to online then ingress returns 404.

Rewrite annotation is not needed in this case as well.

ingress.yaml should look like:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: online-ingress
#  annotations:
spec:
  rules:
    - host: online
      http:
        paths:
          - path: /userOnline
            pathType: Prefix
            backend:
              service:
                name: online
                port:
                  number: 8080

More details about ingress

LoadBalancer on Minikube

Since minikube is considered as bare metal installation, to get external IP for service/ingress, it's necessary to use specially designed metallb solution.

MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.

It ships as add-on for minikube and can be enabled with:

minikube addons enable metallb

And it needs to create a configMap with setup. Please refer to metallb configuration

Minikube Nginx Ingress not finding service endpoint

Answers (1)

Related Questions