Setting cache-control headers for nginx ingress controller on kubernetes GKE

Question

I have an ingress-nginx controller handling traffic to my Kubernetes cluster hosted on GKE. I set it up using helm installation instructions from docs:

Docs here

For the most part everything is working, but if I try to set cache related parameters via a server-snippet annotation, all of the served content that should get the cache-control headers comes back as a 404.

Here's my ingress-service.yaml file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: "4000"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "4000"
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~* \.(js|css|gif|jpe?g|png)$ {
        expires 1M;
        add_header Cache-Control "public";
      }
spec:
  tls:
    - hosts:
        - example.com
      secretName: example-com
  rules:
    - host: example.com
      http:
       paths:
        - path: /
          backend:
            serviceName: client-cluster-ip-service
            servicePort: 5000
        - path: /api/
          backend:
            serviceName: server-cluster-ip-service
            servicePort: 4000

Again, it's only the resources that are matched by the regex that come back as 404 (all .js files, .css files, etc.).

Any thoughts on why this would be happening?

Any help is appreciated!

Answer 1

Configuration-snippet will work with nginx ingress

you can use something like

nginx.ingress.kubernetes.io/configuration-snippet : |
      if ($request_uri ~* \.(js|css|gif|jpe?g|png)) {
        expires 1M;
        add_header Cache-Control "public";
      }

Answer 2

There is another approach to this. You can actually do the ingress way to handle this static content request(dynamically), without editing nginx.conf . Moreover nginx.conf is dynamically created/edited by the ingress-nginx controller every time you make changes in the ingress object. Making changes or mounting the nginx.conf file can create issues.

For the Static files, use the below snippet to redirect your request to the correct service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/server-snippet: |
      expires 1M;
      add_header Cache-Control "public";
spec:
  ingressClassName: nginx
  rules:
  - host: xx.xx.com
      http:
        paths:
        - pathType: Prefix
          path: /status/.*
          backend:
            service:
              name: one-service
              port:
                number: 80
  - host: xx.xx.com
     http:
       paths:
        - pathType: Prefix
          path: /.*(js|css|gif|jpe?g|png)
          backend:
            service:
              name: static-app-service
              port:
                number: 80
  - host: xx.xx.com
      http:
        paths:
        - pathType: Prefix
          path: /
          backend:
            service:
              name: third-service
              port:
                number: 80

Use the nginx.ingress.kubernetes.io/use-regex: "true" for redirecting your static traffic to the correct service.

The above snippet will provide you the results as below:

1. Any request on xx.xx.com/status/ will be routed to one-service.
   More Example :
   xx.xx.com/status/zenduty -> one-service
   xx.xx.com/status/zenduty/incident -> one-service
   xx.xx.com/status/anything -> one-service
2. Any request on xx.xx.com containing .png, .js, .css, .jpg, .gif or .jpeg will be routed to static-app-service
   Example :
   xx.xx.com/custom.js -> static-app-service
   xx.xx.com/status/custom.js -> static-app-service
   xx.xx.com/custom.css -> static-app-service
   xx.xx.com/abc/zen/duty/custom.css -> static-app-service
   xx.xx.com/status/custom.gif -> static-app-service
3. Any other request on xx.xx.com which doesn't contain (.png, .js, .css, .jpg, .gif or .jpeg) and do not start with (xx.xx.com/static/), will be routed to third-service
   Example:
   xx.xx.com/login -> third-service
   xx.xx.com/xyz -> third-service

If you want to add another header to the snippet you can add it. My suggestion will be to create a separate ingress object for static content.

Below is how you can separate the above snippet and add more headers on custom service levels.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/server-snippet: |
      expires 1M;
      add_header Cache-Control "public";
spec:
  ingressClassName: nginx
  rules:
  - host: xx.xx.com
      http:
        paths:
        - pathType: Prefix
          path: /.*(js|css|gif|jpe?g|png)
          backend:
            service:
              name: static-app-service
              port:
                number: 80

The below snippet is for other services which do not have to serve the static content and have custom headers according to their requirements.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/server-snippet: |
      add_header X-XSS-Protection "1";
      add_header Strict-Transport-Security "max-age=31536000;includeSubdomains; preload";
spec:
  ingressClassName: nginx
  rules:
  - host: xx.xx.com
      http:
        paths:
        - pathType: Prefix
          path: /status/.*
          backend:
            service:
              name: one-service
              port:
                number: 80
  - host: xx.xx.com
     http:
       paths:
        - pathType: Prefix
          path: /
          backend:
            service:
              name: third-service
              port:
                number: 80

The ingress class name here is ingressClassName: nginx, please change accordingly.

This is the ingress way to route to the correct service.

If you want to pass client IP to your backend service , Check this for more details : https://stackoverflow.com/questions/65345240/updating-ingress-nginx-controller-configmap-to-pass-client-ip-to-backend-service/71407668#71407668

For Custom error page : How to customize error pages served via the default backend of an nginx ingress controller?

Answer 3

Those location blocks are last and/or longest match wins, and since the ingress itself is not serving any such content, the nginx relies on a proxy_pass directive pointing at the upstream server. Thus, if you are getting 404s, it's very likely because your location is matching, thus interfering with the proxy_pass one. There's a pretty good chance you'd actually want configuration-snippet: instead, likely in combination with if ($request_uri ~* ...) { to add the header.

One can try this locally with a trivial nginx.conf pointing at python3 -m http.server 9090 or whatever fake upstream target.

Separately, for debugging nginx ingress problems, it is often invaluable to consult its actual nginx.conf, which one can grab from any one of the ingress Pods, and/or consulting the logs of the ingress Pods where nginx will emit helpful debugging text.