CODEWITHSUNDEEP
kubernetesnginx-ingress
ozmhsh
ozmhsh

Reputation: 71

404 Not Found error after configuring the Nginx Ingress Controller

UPDATE: The issue persists but I used another way (sub-domain name, instead of the path) to 'bypass' the issue:

ubuntu@df1:~$ cat k8s-dashboard-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: k8s-dashboard-ingress
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/secure-backends: "true"
spec:
  ingressClassName: nginx
  tls: 
    - hosts: 
      - dashboard.XXXX
      secretName: df1-tls
  rules:
  - host: dashboard.XXXX
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443

This error bothers me for some time and I hope with your help I can come down to the bottom of it.

I have one K8S cluster (single node so far, to avoid any network related issues). I installed Grafana on it.

All pods are running fine:

ubuntu:~$ k get po -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
default                grafana-646c8874cb-h6tc5                     1/1     Running   0          11h
default                nginx-1-7bdc99b884-xh7kl                     1/1     Running   0          36h
kube-system            coredns-64897985d-4sk6l                      1/1     Running   0          2d16h
kube-system            coredns-64897985d-dx5h6                      1/1     Running   0          2d16h
kube-system            etcd-df1                                     1/1     Running   1          3d14h
kube-system            kilo-kb52f                                   1/1     Running   0          2d16h
kube-system            kube-apiserver-df1                           1/1     Running   1          3d14h
kube-system            kube-controller-manager-df1                  1/1     Running   4          3d14h
kube-system            kube-flannel-ds-fjkxv                        1/1     Running   0          3d13h
kube-system            kube-proxy-bd2xt                             1/1     Running   0          3d14h
kube-system            kube-scheduler-df1                           1/1     Running   10         3d14h
kubernetes-dashboard   dashboard-metrics-scraper-799d786dbf-5skdw   1/1     Running   0          2d16h
kubernetes-dashboard   kubernetes-dashboard-6b6b86c4c5-56zp2        1/1     Running   0          2d16h
nginx-ingress          nginx-ingress-5b467c7d7-qtqtq                1/1     Running   0          2d15h

As you saw, I installed nginx ingress controller.

Here is the ingress:

ubuntu:~$ k describe ing grafana
Name:             grafana
Labels:           app.kubernetes.io/instance=grafana
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=grafana
                  app.kubernetes.io/version=8.3.3
                  helm.sh/chart=grafana-6.20.5
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host           Path  Backends
  ----           ----  --------
  kalepa.k8s.io  
                 /grafana   grafana:80 (10.244.0.14:3000)
Annotations:     meta.helm.sh/release-name: grafana
                 meta.helm.sh/release-namespace: default
Events:          <none>

Here is the service that is defined in above ingress:

ubuntu:~$ k get svc
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
grafana       ClusterIP   10.96.148.1     <none>        80/TCP     11h
kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP    3d14h

If I do a curl to the cluster ip of the service, it goes through without an issue:

ubuntu:~$ curl 10.96.148.1
<a href="/grafana/login">Found</a>.

If I do a curl to the hostname with the path to the service, I got the 404 error:

ubuntu:~$ curl kalepa.k8s.io/grafana
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.21.5</center>
</body>
</html>

The hostname is resolved to the cluster ip of the nginx ingress service (nodeport):

ubuntu:~$ grep kalepa.k8s.io /etc/hosts
10.96.241.112   kalepa.k8s.io

This is the nginx ingress service definition:

ubuntu:~$ k describe -n nginx-ingress svc nginx-ingress
Name:                     nginx-ingress
Namespace:                nginx-ingress
Labels:                   <none>
Annotations:              <none>
Selector:                 app=nginx-ingress
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.241.112
IPs:                      10.96.241.112
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  31803/TCP
Endpoints:                10.244.0.6:80
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  31913/TCP
Endpoints:                10.244.0.6:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

What am I missing? Thanks for your help!

Upvotes: 1

Views: 6268

Answers (2)

Rakesh Gupta
Rakesh Gupta

Reputation: 3750

This is happening as you are using /grafana and this path does not exist in the grafana application - hence 404. You need to first configure grafana to use this context path before you can forward your traffic to /grafana.

If you use / as path, it will work. That's why curl 10.96.148 works as you are not adding a route /grafana. But most likely that path is already used by some other service, that's why you were using /grafana to begin with.

Therefore, you need to update your grafana.ini file to set the context root explicitly as shown below.

You may put your grafana.ini in a configmap, mount it to the original grafana.ini location and recreate the deployment.

[server]
domain = kalepa.k8s.io
root_url = http://kalepa.k8s.io/grafana/

Upvotes: 3

Bazhikov
Bazhikov

Reputation: 841

I can see there is no ingressClassName specified for your ingress. It looks something like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
spec:
  ingressClassName: nginx 
  tls:
  - hosts:
    - kalepa.k8s.io
    secretName: secret_name
  rules:
  - host: kalepa.k8s.io
    http:
      paths:
         ...

Upvotes: 0

Related Questions