CODEWITHSUNDEEP
kubernetesnginx-ingress
Liogate
Liogate

Reputation: 165

Ingress nginx cache

I'm trying to figure out how to use nginx proxy cache with some specific rules. For exemple, when i'm hosting Ghost or Wordpress, I don't want to cache admin section. Using server snippet, I've tried a lot of different combinaison but still have issues with cache in admin section.

nginx.ingress.kubernetes.io/proxy-buffering: "on"
nginx.ingress.kubernetes.io/server-snippet: |-
  proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
  proxy_ignore_headers Set-Cookie;
  proxy_cache app_cache;
  proxy_cache_lock on;
  proxy_cache_valid any 30m;
  add_header X-Cache-Status $upstream_cache_status;

I want to use nginx code snippet for (ghost|sinout) paths to bypass cache when in admin area, but i'm loosing the proxy_pass context resulting to a 502 bad gateway.

Here is the current ingress config caching every pages, admin path too:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/server-snippet: |-
      proxy_cache my_blog_cache;
      proxy_cache_lock on;
      proxy_cache_valid any 30m;
      add_header X-Cache-Status $upstream_cache_status;
      proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
  name: my-blog
  namespace: web
spec:
  rules:
  - host: blog.example.com
    http:
      paths:
      - backend:
          serviceName: ingress-541322b8660dbd2ceb1e8ff1813f0dd5
          servicePort: 2368
        path: /
  tls:
  - hosts:
    - blog.example.com
    secretName: my-blog-cert
status:
  loadBalancer:
    ingress:
    - ip: 1.2.3.4

Here is the nginx config i'm trying to get but not compatible with ingress annotations:

   location / {
        proxy_cache my_blog_cache;
        proxy_cache_valid 200 30m;
        proxy_cache_valid 404 1m;
        proxy_pass http://ghost_upstream;
        proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
        proxy_ignore_headers Set-Cookie;
        proxy_hide_header Set-Cookie;
        proxy_hide_header X-powered-by;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        expires 10m;
    }
    location /content/images {
        alias /path/to/ghost/content/images;
        access_log off;
        expires max;
    }
    location /assets {
        alias /path/to/ghost/content/themes/uno-master/assets;
        access_log off;
        expires max;
    }
    location /public {
        alias /path/to/ghost/core/built/public;
        access_log off;
        expires max;
    }
    location /ghost/scripts {
        alias /path/to/ghost/core/built/scripts;
        access_log off;
        expires max;
    }
    location ~ ^/(?:ghost|signout) { 
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_pass http://ghost_upstream;
        add_header Cache-Control "no-cache, private, no-store, must-revalidate, max-stale=0, post-check=0, pre-check=0";
    }

Thanks for your help !

Upvotes: 8

Views: 11257

Answers (2)

Artem Zinnatullin
Artem Zinnatullin

Reputation: 4447

I had exact same desire: cache Ghost responses with respect to their Cache-Control headers directly on Nginx Ingress in the Kubernetes cluster.

After some hours spent on it here is my solution:

First step

First of all, you need to define proxy_cache_path on Nginx Ingress ConfigMap level (the docs are really unclear on how to apply it tbh).

In my case, I manage Nginx Ingress installation via Helm, so I've added it to Helm values chart:

# Default values https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml
controller:
  config:
    http-snippet: "proxy_cache_path /tmp/nginx_my_cache levels=1:2 keys_zone=mycache:2m use_temp_path=off max_size=2g inactive=48h;"

Then apply this change:

helm upgrade -f my-nginx-ingress-values.yaml ingress-nginx ingress-nginx/ingress-nginx --recreate-pods

Second step

Now that we have proxy_cache_path set, we need to configure Ingress for particular host with annotations:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myingress
  namespace: mynamespace
  annotations:
    kubernetes.io/ingress.class: "nginx"
    # Buffering must be enabled for Nginx disk cache to work.
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    # See https://www.nginx.com/blog/nginx-caching-guide/
    # Cache Key Zone is configured in Helm config.
    nginx.ingress.kubernetes.io/server-snippet: |
      proxy_cache mycache;
      proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
      proxy_cache_background_update on;
      proxy_cache_revalidate on;
      proxy_cache_lock on;
      add_header X-Cache-Status $upstream_cache_status;

Note:

I spent most time figuring out why I was still getting MISSes. Turned out it's due to nginx.ingress.kubernetes.io/proxy-buffering default in Ingress — off — this DISABLES Nginx caching, thus you have to set it to on which is what we do.

Apply the change to Ingress.

Debugging resulting Nginx config

You can and should I think verify resulting nginx.conf used for Ingress that is generated as result of applying ConfigMap and Ingress-level annotations.

To do so, you can copy nginx.conf from Ingress Controller pod to your local machine and verify its content (or exec into pod and see it there):

# Make sure to use correct namespace where Ingress Controller is deployed
# and correct Ingress Controller Pod name
kubectl cp -n default ingress-nginx-controller-xxxx:/etc/nginx/nginx.conf ~/Desktop/nginx.conf

It should contain all the changes we've made!

Debugging actual response caching

Now that we have everything configured — it's time to verify actual caching. Note that we have added X-Cache-Status header which will indicate if it's a HIT or MISS.

I personally like httpie for HTTP requests from Terminal, you can use curl or browser:

First request is going to be MISS:

http https://example.com/myimage.jpg
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: public, max-age=31536000
Connection: keep-alive
Content-Length: 53588
Content-Type: image/jpeg
Date: Wed, 20 Oct 2021 10:39:06 GMT
ETag: W/"d154-17c3aa43389"
Last-Modified: Fri, 01 Oct 2021 06:56:52 GMT
Strict-Transport-Security: max-age=15724800; includeSubDomains
X-Cache-Status: HIT
X-Powered-By: Express
X-Request-ID: 0c73f97cb51d3071f14968720a26a99a

+-----------------------------------------+
| NOTE: binary data not shown in terminal |
+-----------------------------------------+

Second request to same URL is now a HIT and doesn't hit the actual Ghost installation, success!

http https://example.com/myimage.jpg
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: public, max-age=31536000
Connection: keep-alive
Content-Length: 53588
Content-Type: image/jpeg
Date: Wed, 20 Oct 2021 10:39:43 GMT
ETag: W/"d154-17c3aa43389"
Last-Modified: Fri, 01 Oct 2021 06:56:52 GMT
Strict-Transport-Security: max-age=15724800; includeSubDomains
X-Cache-Status: HIT
X-Powered-By: Express
X-Request-ID: 0c73f97cb51d3071f14968720a26a99a

+-----------------------------------------+
| NOTE: binary data not shown in terminal |
+-----------------------------------------+

It's also useful to verify logs on Ghost to double-check that cache HIT requests actually served directly from Nginx and never hit Ghost.

Upvotes: 8

Mr.KoopaKiller
Mr.KoopaKiller

Reputation: 3982

There are some options in nginx-ingress controller that is only possible to change using ConfigMap and other is possible using Annnotations, as you are doing.

You can combine both to reach the expected result, or create a custom template.

Here you can see an alternative that maybe could help you.

Upvotes: 2

Related Questions