Varnish caching wordpress when not instructed

Question

I am trying to write a good vcl file for my sites that use a very customized Wordpress installation, but I am still a newbie in the field. So I will be very thankful if anyone can point out where I make mistakes. I am trying to have a default.vcl with common rules and per domain vcl files with domain specific rules. The problem is that I try to instruct Varnish not to cache some paths for pages and pop-up ajax boxes. And not to cache logged in users which I catch by login cookie.

I shall paste my complete config. Also, I am bit confused about the linear execution of the vcl file if there is an included additional vcl file. Does the location where is the 'includ' matter? I would like to investigate the case myself, but using Varnish logging is a bit complicated. Is there a way that I can find which rules are acting on specific pages?

My default.vcl

vcl 4.0;

import std;

backend domain1 {
    .host = "11.111.11.1";
    .port = "8001";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .max_connections = 800;
    }

backend domain2 {
    .host = "222.22.22.2";
    .port = "8002";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .max_connections = 800;
    }

    acl purge {
                "localhost";
                "127.0.0.1";
                "79.124.64.16";
                "193.107.37.45";

            }


sub vcl_recv {
    if (req.http.host ~ "^(?i)domain1.com" || req.http.host ~ "^.*\.domain1.com" ) {
        set req.backend_hint = domain1;
        }
    elseif  (req.http.host == "^.*\.domain2.com" || req.http.host == "^.*\.domain2-alias.com") {
        set req.backend_hint = domain2;
        }



    if (req.restarts == 0) {
        if (req.http.X-Forwarded-For) {
            set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
        } else {
            set req.http.X-Forwarded-For = client.ip;
            }
        }


    #Pass through cache PhpMyAdmin
    if (req.url ~ "^.*phpmyadmin.*") {
        return(pass);
            }
        }

######## PER DOMAIN SUBROUTINES ###########

include "domain1.vcl";

include "domain2.vcl";

    if (req.http.Cookie ~ "wordpress_logged_in_") {
        return (pass);
        }



    # Normalize the header, remove the port (in case you're testing this on various TCP ports)
    set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

    # Allow purging from ACL
    if (req.method == "PURGE") {
        # If not allowed then a error 405 is returned
        if (!client.ip ~ purge) {
            return(synth(405, "This IP is not allowed to send PURGE requests."));
            }

        # If allowed, do a cache_lookup -> vlc_hit() or vlc_miss()
            return (purge);
        }

    # Post requests will not be cached
        if (req.http.Authorization || req.method == "POST") {
            return (pass);
            }


    # Conflicting with few lines below
    #    # Only deal with "normal" types
    #    if (req.method != "GET" &&
    #        req.method != "HEAD" &&
    #        req.method != "PUT" &&
    #        req.method != "POST" &&
    #        req.method != "TRACE" &&
    #        req.method != "OPTIONS" &&
    #        req.method != "PATCH" &&
    #        req.method != "DELETE") {

    #    /* Non-RFC2616 or CONNECT which is weird. */
    #    return (pipe);
    #    }


    # Implementing websocket support (https://www.varnish-cache.org/docs/4.0/users-guide/vcl-example-websockets.html)
    if (req.http.Upgrade ~ "(?i)websocket") {
        return (pipe);
        }


    # Only cache GET or HEAD requests. This makes sure the POST requests are always passed.
    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
        }


    # Some generic URL manipulation, useful for all templates that follow
    # First remove the Google Analytics added parameters, useless for our backend
    if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=") {
        set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
        set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
        set req.url = regsub(req.url, "\?&", "?");
        set req.url = regsub(req.url, "\?$", "");
        }

    # Some generic cookie manipulation, useful for all templates that follow
    # Remove the "has_js" cookie
    set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");

    # Remove any Google Analytics based cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", "");

    # Remove DoubleClick offensive cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");

    # Remove the Quant Capital cookies (added by some plugin, all __qca)
    set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");

    # Remove the AddThis cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");

    # Remove a ";" prefix in the cookie if present
    set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");

    #Remove WP cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__utma.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "__utmb.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "__utmc.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "__utmt.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "__utmz.=[^;]+(; )?", "");
    #    set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie.=[^;]+(; )?", "");
    #    set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "bp-message[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "bp-message-type[^;]+(; )?", "");


    # Are there cookies left with only spaces or that are empty?
    if (req.http.cookie ~ "^\s*$") {
        unset req.http.cookie;
        }

    **#Dont cache Logged-in users
        if (req.http.Cookie == "wordpress_logged_in_") {
            return (pass);
        }
        elseifif (req.http.cookie ~ "wordpress_logged_in_") {
            return (pass);
        }**

    # Large static files are delivered directly to the end-user without
    # waiting for Varnish to fully read the file first.
    # Varnish 4 fully supports Streaming, so set do_stream in vcl_backend_response()
    if (req.url ~ "^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|rar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") {
        unset req.http.Cookie;
            return (hash);
        }


    # Remove all cookies for static files
    # A valid discussion could be held on this line: do you really need to cache static files that don't cause load? Only if you have memory left.
    # Sure, there's disk I/O, but chances are your OS will already have these files in their buffers (thus memory).
    # Before you blindly enable this, have a read here: https://ma.ttias.be/stop-caching-static-files/
    if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|
    woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
        unset req.http.Cookie;
            return (hash);
        }

    # Send Surrogate-Capability headers to announce ESI support to backend
    #    set req.http.Surrogate-Capability = "key=ESI/1.0";

    if (req.http.Authorization) {
        # Not cacheable by default
        return (pass);
        }

    # Did not cache HTTP authentication and HTTP Cookie
    if (req.http.Authorization || req.http.Cookie) {
        # Not cacheable by default
        return (pass);
        }

    }
# Cache all others requests
return (hash);

}


######## PER DOMAIN SUBROUTINES ###########

include "domain1.vcl";

include "domain2.vcl";




sub vcl_pipe {

    # set bereq.http.Connection = "Close";
    # Implementing websocket support (https://www.varnish-cache.org/docs/4.0/users-guide/vcl-example-websockets.html)
    if (req.http.upgrade) {
        set bereq.http.upgrade = req.http.upgrade;
        }
    return (pipe);
}



sub vcl_pass {

    # return (pass);

}


# The data on which the hashing will take place
sub vcl_hash {

    hash_data(req.url);
        if (req.http.host) {
            hash_data(req.http.host);
        } else {
            hash_data(server.ip);
        }

    # hash cookies for requests that have them
    #    if (req.http.Cookie) {
    #       hash_data(req.http.Cookie);
    #    }
    }



sub vcl_hit {

    if (obj.ttl >= 0s) {
    # A pure unadultered hit, deliver it
        return (deliver);
        }

    # https://www.varnish-cache.org/docs/trunk/users-guide/vcl-grace.html
    # When several clients are requesting the same page Varnish will send one request to the backend and place the others on hold while fetching one copy from the backend. In some products this is called request coalescing and Varni
    sh does this automatically.
    # If you are serving thousands of hits per second the queue of waiting requests can get huge. There are two potential problems - one is a thundering herd problem - suddenly releasing a thousand threads to serve content might sen
    d the load sky high. Secondly - nobody likes to wait. To deal with this we can instruct Varnish to keep the objects in cache beyond their TTL and to serve the waiting requests somewhat stale content.

    # if (!std.healthy(req.backend_hint) && (obj.ttl + obj.grace > 0s)) {
    #   return (deliver);
    # } else {
    #   return (fetch);
    # }

    # We have no fresh fish. Lets look at the stale ones.
    ##if (std.healthy(req.backend_hint) {
    # Backend is healthy. Limit age to 10s.
    ##if (obj.ttl + 10s > 0s) {
            #set req.http.grace = "normal(limited)";
    ##          return (deliver);
    ##    } else {
        # No candidate for grace. Fetch a fresh object.
    ##          return(fetch);
    ##          }
    ##    } else {
        # backend is sick - use full grace
    ##    if (obj.ttl + obj.grace > 0s) {
        #set req.http.grace = "full";
    ##          return (deliver);
    ##      } else {
            # no graced object.
                return (fetch);
    ##      }
    ##    }

        # fetch & deliver once we get the result
    ##    return (fetch); # Dead code, keep as a safeguard
    }


sub vcl_miss {
    # Called after a cache lookup if the requested document was not found in the cache. Its purpose
    # is to decide whether or not to attempt to retrieve the document from the backend, and which
    # backend to use.

    return (fetch);
}


# Handle the HTTP request coming from our backend
    sub vcl_backend_response {
        # Called after the response headers has been successfully retrieved from the backend.

         if (bereq.http.Cookie ~ "(UserID|_session)") {
             set beresp.http.X-Cacheable = "NO:Got Session";
             set beresp.uncacheable = true;
                 return (deliver);

         } elsif (beresp.ttl <= 0s) {
             # Varnish determined the object was not cacheable
            set beresp.http.X-Cacheable = "NO:Not Cacheable";

         } elsif (beresp.http.set-cookie) {
            # You don't wish to cache content for logged in users
            set beresp.http.X-Cacheable = "NO:Set-Cookie";
            set beresp.uncacheable = true;
                 return (deliver);

         } elsif (beresp.http.Cache-Control ~ "private") {
            # You are respecting the Cache-Control=private header from the backend
            set beresp.http.X-Cacheable = "NO:Cache-Control=private";
            set beresp.uncacheable = true;
                 return (deliver);
         } else {
         # Varnish determined the object was cacheable
            set beresp.http.X-Cacheable = "YES";
        }



        # Pause ESI request and remove Surrogate-Control header
        if (beresp.http.Surrogate-Control ~ "ESI/1.0") {
              unset beresp.http.Surrogate-Control;
              set beresp.do_esi = true;
        }

        # Enable cache for all static files
        # The same argument as the static caches from above: monitor your cache size, if you get data nuked out of it, consider giving up the static file cache.
        # Before you blindly enable this, have a read here: https://ma.ttias.be/stop-caching-static-files/
        if (bereq.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|wof
    f|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
            unset beresp.http.set-cookie;
        }

        # Large static files are delivered directly to the end-user without
        # waiting for Varnish to fully read the file first.
        # Varnish 4 fully supports Streaming, so use streaming here to avoid locking.
        if (bereq.url ~ "^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|rar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") {
            unset beresp.http.set-cookie;
            set beresp.do_stream = true;  # Check memory usage it'll grow in fetch_chunksize blocks (128k by default) if the backend doesn't send a Content-Length header, so only enable it for big objects
        }

        # Sometimes, a 301 or 302 redirect formed via Apache's mod_rewrite can mess with the HTTP port that is being passed along.
        # This often happens with simple rewrite rules in a scenario where Varnish runs on :80 and Apache on :8080 on the same box.
        # A redirect can then often redirect the end-user to a URL on :8080, where it should be :80.
        # This may need finetuning on your setup.
        #
        # To prevent accidental replace, we only filter the 301/302 redirects for now.
        if (beresp.status == 301 || beresp.status == 302) {
            set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+", "");
        }

        # Set 2min cache if unset for static files
        if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") {
            set beresp.ttl = 120s; # Important, you shouldn't rely on this, SET YOUR HEADERS in the backend
            set beresp.uncacheable = true;
            return (deliver);
        }

        # Don't cache 50x responses
        if (beresp.status == 500 || beresp.status == 502 || beresp.status == 503 || beresp.status == 504) {
             return (abandon);
        }

        # Allow stale content, in case the backend goes down.
        # make Varnish keep all objects for 6 hours beyond their TTL
        set beresp.grace = 6h;

        return (deliver);
    }

    # The routine when we deliver the HTTP request to the user
    # Last chance to modify headers that are sent to the client

    sub vcl_deliver {
        # Called before a cached object is delivered to the client.

        if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed
            set resp.http.X-Cache = "HIT";
         } else {
         set resp.http.X-Cache = "MISS";
        }

        # Please note that obj.hits behaviour changed in 4.0, now it counts per objecthead, not per object
        # and obj.hits may not be reset in some cases where bans are in use. See bug 1492 for details.
        # So take hits with a grain of salt
        #set resp.http.X-Cache-Hits = obj.hits;

        # Remove some headers: PHP version
        unset resp.http.X-Powered-By;

        # Remove some headers: Apache version & OS
        unset resp.http.Server;
        unset resp.http.X-Drupal-Cache;
        unset resp.http.X-Varnish;
        unset resp.http.Via;
        unset resp.http.Link;
        unset resp.http.X-Generator;

        return (deliver);
    }


    sub vcl_purge {
        # Only handle actual PURGE HTTP methods, everything else is discarded
        if (req.method != "PURGE") {
            # restart request
            set req.http.X-Purge = "Yes";
            return(restart);
         }
    }


    sub vcl_synth {
        if (resp.status == 720) {
        # We use this special error status 720 to force redirects with 301 (permanent) redirects
        # To use this, call the following from anywhere in vcl_recv: return (synth(720, "http://host/new.html"));
        set resp.http.Location = resp.reason;
        set resp.status = 301;
            return (deliver);
        } elseif (resp.status == 721) {
        # And we use error status 721 to force redirects with a 302 (temporary) redirect
        # To use this, call the following from anywhere in vcl_recv: return (synth(720, "http://host/new.html"));
        set resp.http.Location = resp.reason;
        set resp.status = 302;
            return (deliver);
        }

            return (deliver);
    }

    sub vcl_fini {
        # Called when VCL is discarded only after all requests have exited the VCL.
        # Typically used to clean up VMODs.

        return (ok);
}

My domain1.vcl

vcl 4.0;

sub vcl_recv {
    if (req.backend_hint == domain1) {


        **#Don't store backend in case of login pop screen
        if (req.url ~ "^.*wp-(login|admin).*" || req.url ~ "preview=true" || req.url ~ "^.*(login|admin).*" ||  req.url ~ "^.*wp-load.*") {
            return (pipe);
        }


        # Do not cache home, search, registration pages
        if (req.url ~ "\/path\/subpath\/" || req.url ~ "\/pathabc\/pathdef\/" || req.url ~ ".*register.*") {
            return(pipe);
        }

        if (req.url ~ ".*\/members\/.*") {
            return(hash);
        }

    }

    return(hash);
}**

sub vcl_backend_response {

}

sub vcl_deliver {
}

My domain2.vcl experiences no issues when creating rules.

Answer 1

The problem you have is due to req.backend_hint == domain1 (this will not work). You should compare with hostname in your site specific VCL instead. So in domain1.vcl it will be:

vcl 4.0;

sub vcl_recv {
    if (req.http.host ~ "^(?i)domain1.com" || req.http.host ~ "^.*\.domain1.com" ) {
       # site specific logic goes here ...
    }
}

You should not place include directives directly inside vcl_recv procedure. Those should go either at the very top of the file or at the bottom depending on desired priority.

In this case you should place them at the top which means :

# ...
include "domain1.vcl";
include "domain2.vcl";

sub vcl_recv {
#...

This will make sure that vcl_recv logic that is defined in your site specific VCL files will have priority in execution over the ones that you have in the main VCL file itself.

The VCL includes work in a way that every procedure in further included file will be executed last. As example, the vcl_recv routines will be executed in this order:

1. vcl_recv from site-specific .vcl
2. vcl_recv from default.vcl
3. vcl_recv from builtin.vcl

More info about writing site specific VCL and what builtin.vcl is you can find here.

Your VCL is unnecessarily complex. I always advise to start from the empty default.vcl that is shipped with Varnish and add more code slowly bit by bit in order to understand how things work. Blind copy pasting never leads to expected results, more often to unexpected :).

Examples

The following is completely unnecessary. Varnish 4 handles this for you before any VCL is executed in the first place:

if (req.restarts == 0) {
    if (req.http.X-Forwarded-For) {
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    } else {
        set req.http.X-Forwarded-For = client.ip;
        }
    }
}

Furthermore, having 600s timeout values for everything would maybe make your application work, but if it doesn't work with the default values (surely enough you'd see "backend fetch failed") there is a reason for that. And Varnish is not the place to fix "backend fetch failed" - your application is :).

Finally, if you have decided to implement your caching policy based on presence of the Wordpress cookies, then stick to that and remove all the lines that filter out the known unneeded cookies. Those are unnecessary (think why would you do whitelisting and blacklisting of cookies at the same time).

I am referring to all lines similar to this:

set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");
# ...