CODEWITHSUNDEEP
varnishvarnish-vclvarnish-4
Michael
Michael

Reputation: 4461

Make varnish serve request from cache (cookies are cleaned)

HTML

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
    <!-- Global site tag (gtag.js) - Google Analytics -->
    <script async src="https://www.googletagmanager.com/gtag/js?id=UA-54516992-1"></script>
    <script>
      window.dataLayer = window.dataLayer || [];
      function gtag(){dataLayer.push(arguments);}
      gtag('js', new Date());

      gtag('config', 'UA-54516992-1');
    </script>

</head>
<body>
    test
</body>
</html>

Backend. This is Python + Django. Please, don't be afraid. I just set a cookie here (render_to_response) and then I need a place to stop at a breakpoint (below in the code it is shown by a comment where it is).

class HomeView(TemplateView):
    template_name = "home/home.html"

    def render_to_response(self, context, **response_kwargs):
        response = super(TemplateView, self).render_to_response(context, **response_kwargs)
        response.set_cookie('sort_on', 'title')
        return response

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        return context # Breakpoint.

varnishd For learning purposes I just clean all cookies.

$ varnishd -V
varnishd (varnish-6.0.6 revision 29a1a8243dbef3d973aec28dc90403188c1dc8e7)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2019 Varnish Software AS

VCL

vcl 4.1;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    call remove_proprietary_cookies;

    if (req.http.Cookie) {      
        return (pass);
    }

}   


sub remove_proprietary_cookies{
    set req.http.Cookie = regsuball(req.http.Cookie, ".*", "");
}

Then I reload the page a couple of times for the cookie to be definitely there.

In Chrome:

document.cookie
"sort_on=title; _ga=GA1.1.988164671.1586849704; _gid=GA1.1.995393496.1586849704; _gat_gtag_UA_54516992_1=1"

Picture of cookies (duplicates of the text above):

enter image description here

Ok. We've checked that Cookie is set. Now let's check that the cookie is properly cleaned.

I stop at the braakpoint and check cookie. The value is {}.

enter image description here

Well. All cookies seems to be cleaned.

The problem: When reloading I constantly come to the breakpoint. This means that Varnish passes the request to the backend. In other words if (req.http.Cookie) not working as I expect. I expect that at the previous step I have removed cookies. Then I check if cookies exist. And there shouldn't be any.

Could you help me:

  1. Understand what is going on here.

  2. Organize everything so that I should definitely remove cookies if I have removed them incorrectly.

  3. Make Varnish serve this request from cache without passing it to the backend.

==============ADDED ON 16th of APRIL================= +++++++++++++++++++++++++++++++++++++++++++++++++++++

I have upgraded Varnish to 6.4:

michael@michael:~$ varnishd -V
varnishd (varnish-6.4.0 revision 13f137934ec1cf14af66baf7896311115ee35598)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2020 Varnish Software AS

What we are going to test:

  1. Set Google Analytics cookies and a cookie at the backend.
  2. Let Varnish remove all cookies and cache the response.
  3. Check that request is passed to the backend only once.

Then I organized nginx behind Varnish (only to log requests). Nginx listens at 8090. This is log config:

   log_format  main  '[$time_local] $http_cookie';

Cookies:

  1. In HTML I organize Google Analytics' tracking.
  2. At the backend I set cookie called "sort_on" with value "title".

+++++++++++++++++++++++++++++++++++++++++++++++++++++++

If we don't cut cookies, the log looks this (this is just for COMPARISON):

127.0.0.1 - - [16/Apr/2020:08:11:05 +0300] "GET / HTTP/1.1" sort_on=title; _ga=GA1.1.236170900.1587013419; _gid=GA1.1.2033785209.1587013419 200 334 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36" "127.0.0.1"

+++++++++++++++++++++++++++++++++++++++++++++++++++++++

varnish.ctl

vcl 4.1;
import cookie;

backend default {
    .host = "127.0.0.1";
    .port = "8090";
}


sub vcl_recv {
    unset req.http.Cookie;
    if (req.http.Cookie) {  
        return (pass);
    }   
}

Nginx passes requests to the backend. Backend is at port 8080.

In short what listens where:

Varnish - 8000
Nginx - 8090
Backend - 8080

Start varnishd:

michael@michael:~$ sudo varnishd -f /home/michael/PycharmProjects/varnish/varnish.vcl -a localhost:8000

Open Chrome, load the page several times:

enter image description here

Nginx's access log:

127.0.0.1 - - [16/Apr/2020:08:12:49 +0300] "GET / HTTP/1.1" - 200 334 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36" "127.0.0.1"
127.0.0.1 - - [16/Apr/2020:08:13:21 +0300] "GET / HTTP/1.1" - 200 334 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36" "127.0.0.1"

We can see that cookie is absent. It is Ok. Cookies are cut off.

Problem: Requests are inevitably passed to the backend. I always stop at the breakpoint.

enter image description here

Will you be so kind as to give me a piece of advice on how to cope with the problem?

Upvotes: 0

Views: 537

Answers (1)

Thijs Feryn
Thijs Feryn

Reputation: 4808

Varnish doesn't look at the contents of the Cookie request header, but rather checks if the header is present.

What you need to do is to check if the Cookie header is empty, and if so, just remove the entire thing.

Just add this after your regsub statement:

if (req.http.Cookie ~ "^\s*$") {
    unset req.http.Cookie;
}

Make the process a bit more flexible

In reality you'll probably will not remove all cookies, but just the ones that are not essential to your application.

You can use this statement to remove all cookies, except that ones you need for your application:

sub vcl_recv {
  if (req.http.Cookie) {
    set req.http.Cookie = ";" + req.http.Cookie;
    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
    set req.http.Cookie = regsuball(req.http.Cookie, 
        ";(SESS[a-z0-9]+|NO_CACHE)=", "; \1=");
    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

    if (req.http.Cookie ~ "^\s*$") {
      unset req.http.Cookie;
    }
    else {
      return (pass);
    }
  }
}

This snippet will remove all cookies, expect that ones that match the SESS[a-z0-9]+|NO_CACHE regex.

Use vmod_cookie

There is a cleaner way to approach cookie substitution in Varnish and it involves using the vmod_cookie module in Varnish. You can find it here: https://github.com/varnishcache/varnish-cache/tree/master/lib/libvmod_cookie

If you upgrade to Varnish version 6.4, the vmod_cookie will be part of the core Varnish installation.

This is the equivalent using vmod_cookie:

vcl 4.1;
import cookie;

sub vcl_recv {
    cookie.parse(req.http.cookie);
    cookie.keep_re("SESS[a-z0-9]+,NO_CACHE");
    set req.http.cookie = cookie.get_string();
    if (req.http.cookie ~ "^\s*$") {
        unset req.http.cookie;
    }
}

Upvotes: 4

Related Questions