Ingress nginx giving 404

Question

I have a microk8s single node K8 cluster. And I am not installing nginx using the microk8s enable ingress command.

I went the native route and installed the helm chart that the nginx site has.

More setup info:

Linux machine, Alma linux running K8 via microk8s

Current pods

[root@node-3 files]# kubectl get pods -owide
NAME                                                         READY   STATUS    RESTARTS   AGE    IP            NODE     NOMINATED NODE   READINESS GATES
apple-app                                                    1/1     Running   0          146m   10.1.139.73   node-3   <none>           <none>
banana-app                                                   1/1     Running   0          164m   10.1.139.72   node-3   <none>           <none>
my-release-nginx-nginx-ingress-controller-7fbc5fc7db-rx26x   1/1     Running   0          15m    10.1.139.74   node-3   <none>           <none>
[root@node-3 files]#

Current services

[root@node-3 files]# kubectl get svc -owide
NAME                                        TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE     SELECTOR
apple-service                               ClusterIP      10.152.183.144   <none>          5678/TCP                     164m    app=apple
banana-service                              ClusterIP      10.152.183.54    <none>          5678/TCP                     164m    app=banana
kubernetes                                  ClusterIP      10.152.183.1     <none>          443/TCP                      3h33m   <none>
my-release-nginx-nginx-ingress-controller   LoadBalancer   10.152.183.112   192.168.1.200   80:32446/TCP,443:31976/TCP   15m     app.kubernetes.io/instance=my-release-nginx,app.kubernetes.io/name=nginx-ingress
[root@node-3 files]#

IP address of the Linux machine: 192.168.1.103

I edited the service of type loadbalnacer that was created via helm for ingress and added the config for ip address to yaml manually

The yaml looks like below

[root@node-3 files]# kubectl get svc my-release-nginx-nginx-ingress-controller -oyaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: my-release-nginx
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2025-03-01T23:15:45Z"
  labels:
    app.kubernetes.io/instance: my-release-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx-ingress
    app.kubernetes.io/version: 4.0.1
    helm.sh/chart: nginx-ingress-2.0.1
  name: my-release-nginx-nginx-ingress-controller
  namespace: default
  resourceVersion: "17851"
  uid: 8ac9eac5-f6d3-440a-9aa8-c473110824bf
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.152.183.112
  clusterIPs:
  - 10.152.183.112
  externalIPs:
  - 192.168.1.200
  externalTrafficPolicy: Local
  healthCheckNodePort: 31646
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    nodePort: 32446
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    nodePort: 31976
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app.kubernetes.io/instance: my-release-nginx
    app.kubernetes.io/name: nginx-ingress
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

The way apple-app pod is, anything that goes to it, it echoes back the word apple. I have tried it via port forwarding and it works fine.

My ingress looks like below

[root@node-3 files]# kubectl get ingress test-ingress -oyaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  creationTimestamp: "2025-03-02T00:22:39Z"
  generation: 1
  name: test-ingress
  namespace: default
  resourceVersion: "23168"
  uid: ad36c8d7-215e-4f79-85c4-11b5251bb18c
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: apple-service
            port:
              number: 5678
        path: /apple
        pathType: Prefix
status:
  loadBalancer: {}
[root@node-3 files]#

But when I try to run the below, it fails

[root@node-3 files]# curl 192.168.1.200/apple
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.27.4</center>
</body>
</html>
[root@node-3 files]#

When I exec into the ingress controller pod and try the curl directly, it does work fine as well

nginx@my-release-nginx-nginx-ingress-controller-7fbc5fc7db-rx26x:/$ curl apple-service:5678/apple
apple
nginx@my-release-nginx-nginx-ingress-controller-7fbc5fc7db-rx26x:/$