CODEWITHSUNDEEP
kuberneteshttpskubernetes-ingressamazon-eksnginx-ingress
user2725290
user2725290

Reputation: 33

TLS nginx ingress in AWS EKS Cluster results in 404 Not Found

I am trying to use Kubernetes Ingress Nginx Controller and running a simple nginx server in AWS EKS.

Browser (https) --> Route 53 (DNS) --> CLB --> nginx Ingress (Terminate TLS) --> Service --> POD

But I am receiving 404 error in browser (url used: https://example.com/my-nginx):

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.10</center>
</body>
</html>

and in ingress logs (kubectl logs -n nginx-ingress nginx-ingress-nginx-controller-6db6f85bc4-mfpwx), I can see below:

192.168.134.181 - - [24/Apr/2021:19:02:01 +0000] "GET /my-nginx HTTP/2.0" 404 154 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0" 219 0.002 [eshop-dev-my-nginx-9443] [] 192.168.168.105:80 154 0.000 404 42fbe692a032bb40bf193954526369cd

Here is my deployment yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
  namespace: eshop-dev
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80

Service yaml:

apiVersion: v1
kind: Service
metadata:
  namespace: eshop-dev
  name: my-nginx
spec:
  selector:
    run: my-nginx
  ports:
    - name: server
      port: 9443
      targetPort: 80
      protocol: TCP

and ingress yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: eshop-dev
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /my-nginx
        pathType: ImplementationSpecific
        backend:
          service:
            name: my-nginx
            port:
                number: 9443
  tls:
  - hosts:
    - example.com
    secretName: externaluicerts

I have verified that service returns the desired output, when used with port forwarding:

kubectl -n eshop-dev port-forward service/my-nginx 9443:9443

I'm not sure if the ingress is incorrectly configured or if it is another problem.Thanks in advance for the help!

nginx-port-forward

Here is the output of kubectl get ingress -n eshop-dev test-ingress -o yaml

kubectl get ingress -n eshop-dev test-ingress -o yaml
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"test-ingress","namespace":"eshop-dev"},"spec":{"rules":[{"host":"example.com","http":{"paths":[{"backend":{"service":{"name":"my-nginx","port":{"number":9443}}},"path":"/my-nginx","pathType":"ImplementationSpecific"}]}}],"tls":[{"hosts":["example.com"],"secretName":"externaluicerts"}]}}
    kubernetes.io/ingress.class: nginx
  creationTimestamp: "2021-04-24T13:16:21Z"
  generation: 13
  managedFields:
  - apiVersion: networking.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:loadBalancer:
          f:ingress: {}
    manager: nginx-ingress-controller
    operation: Update
    time: "2021-04-24T13:16:40Z"
  - apiVersion: extensions/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-04-24T13:18:36Z"
  - apiVersion: networking.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:kubernetes.io/ingress.class: {}
      f:spec:
        f:rules: {}
        f:tls: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-04-24T16:33:47Z"
  name: test-ingress
  namespace: eshop-dev
  resourceVersion: "7555944"
  selfLink: /apis/extensions/v1beta1/namespaces/eshop-dev/ingresses/test-ingress
  uid: a7694655-20c6-48c7-8adc-cf3a53cf2ffe
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: my-nginx
          servicePort: 9443
        path: /my-nginx
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - example.com
    secretName: externaluicerts
status:
  loadBalancer:
    ingress:
    - hostname: xxxxxxxxxxxxxxxxdc75878b2-433872486.eu-west-1.elb.amazonaws.com

Upvotes: 2

Views: 4734

Answers (1)

AndD
AndD

Reputation: 2701

From the image you posted of the nginx-port-forward, I see you went on localhost:9443 directly, which means the Nginx server you are trying to access serve its content under /

But in the ingress definition, you define that the service will be served with path: /my-nginx. This could be the problem, as you are requesting https://example.com/my-nginx which will basically go to my-nginx:9443/my-nginx and, depending on the Pod behind this service, it could return a 404 if there's nothing at that path.

To test if the problem is what I specified above, you have a few options:

  • easiest one, remove path: /my-nginx an, instead, go with path: /. You could also specify pathType: Prefix which means that everything matching the subPath specified will be served by the service.
  • Add a rewrite target, which is necessary if you want to serve a service at a different path from the one expected by the application.

Add an annotation similar to the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: eshop-dev
  annotations:
    kubernetes.io/ingress.class: "nginx"
    # this will rewrite request under / + second capture group
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: example.com
    http:
      paths:
        # this will serve all paths under /my-nginx and capture groups for regex annotations
      - path: /my-nginx(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: my-nginx
            port:
              number: 9443
  tls:
  - hosts:
    - example.com
    secretName: externaluicerts
  • Configure your application to know that it will be served under the path you desire. This is often the better approach, as frontend applications should almost always be served under the path that they expect to be.

From the info you posted, I think this is the problem an once fixed, your setup should work.

If you are curious about rewrite targets or how paths work in an Ingress, here is some documentation:

Rewrites ( https://kubernetes.github.io/ingress-nginx/examples/rewrite/#rewrite )

Path types ( https://kubernetes.io/docs/concepts/services-networking/ingress/#path-types )

Update

About why configuring the application to directly serve its content at the path specified in the Ingress (basically to know at which path it will served) is the best solution:

Let's say you serve a complex application in your Pod which will serve its content under /. The main page will try to load several other resources like css, js code and so on, everything from the root directory. Basically, if I open /, the app will load also:

/example.js
/my-beautiful.css

Now, if I serve this app behind an ingress at another path, let's say under /test/ with a rewrite target, the main page will work, because:

/test/ --> /  # this is my rewrite rule

but then, the page will request /example.js, and the rewrite works in one direction only, so the browser will request a resource which will go in 404, because the request should have been /test/example.js (as that would rewrite to remove the /test part of the path)

So, with frontend applications, rewrite targets may not be enough, mostly if the applications request resources with absolute paths. With just REST API or single requests instead, rewrites usually works great.

Upvotes: 2

Related Questions