Varnish 7.4.2 VARY cookie and compression problem

Question

I use on my servers varnish 7.1.2 and I want to upgrade to varnish 7.4.2 Everything is OK but since I've make the upgrade, I've noticed a strange behaviour: When I made a curl, I see that VARY cookie is changed from:

vary: Accept-Encoding, Cookie, User-Agent - CORRECT

to

vary: User-Agent, Cookie - WRONG

This translates into a huge increasing in network usage, becouse it seems that the content is no more compressed both when varnish gets it from the backend, and in the communication between the client and varnish.

After a lot of searching, I've added to function vcl_backend_response this:

  if (beresp.http.content-type ~ "text") {
    set beresp.do_gzip = true;
  }

The situation seems improving, as the client receives the compressed content and the "accept-encoding" appears again on the vary cookie. However, it seems that when varnish fetches content from the backend, it is always uncompressed and therefore network occupancy still remains high.

For the moment I have decided to roll back to 7.1.2 but it is still not a sustainable solution in the long term.

Can anyone help me understand?

Thanks a lot

Answer 1

Hello and many many thanks for the answer.

My backend is haproxy+nginx. I think everithing works OK with the backend, because if I try a curl direct on haproxy with the Header Accept-Encoding:gzip, I receive:

curl -k -I https://X.X.X.X/foo/bar -H"Host:foo.bar" -H 'Accept-Encoding:gzip'
HTTP/2 200
server: nginx
content-type: text/html; charset=utf-8
vary: Accept-Encoding
vary: Cookie
date: Fri, 01 Dec 2023 17:00:55 GMT
expires: Fri, 01 Dec 2023 17:06:45 GMT
cache-control: public, max-age=350, post-check=350, pre-check=350
x-xss-protection: 1
x-fastcgi-cache: MISS
content-encoding: gzip
strict-transport-security: max-age=63072000; includeSubDomains; preload
x-content-type-options: nosniff

As you can see it's present the content-encoding: gzip

Anyway, I've tried to add to nginx on a test environment the option gzip_proxied any; also with varnish 7.4.2 the things seems good.

This is the CURL on varnish 7.4.2 that point to a backend HAPROXY+NGINX where I've enabled on nginx the option you suggest:

curl -k -I https://X.X.X.X/foo/bar.html -H"Host:www.foo.bar" -H 'Accept-Encoding:gzip'
HTTP/2 200
server: nginx
content-type: text/html; charset=utf-8
date: Fri, 01 Dec 2023 17:24:35 GMT
expires: Fri, 01 Dec 2023 17:29:41 GMT
cache-control: public, max-age=306, post-check=306, pre-check=306
x-xss-protection: 1
x-fastcgi-cache: MISS
content-encoding: gzip
x-varnish-backend: haproxy-test
x-ua-device: desktop
x-varnish: 65538
age: 0
via: 1.1 test-varnish (Varnish/7.4)
accept-ranges: bytes
vary: Accept-Encoding, Cookie, User-Agent
x-varnish-beresp: 200
x-varnsih-cache: MISS
remote-ip: 1.2.3.4
server-hostname: test-varnish
strict-transport-security: max-age=63072000; includeSubDomains; preload
x-content-type-options: nosniff
alt-svc: h3=":443"; ma=60

It seems good! Next week I'll try to put the change in production environment (now it's Friday, you know :-D) and we can see how it works.

Also the command for the log debug has been very useful. Thanks a lot. I'll update the thread next week.

Bye

Answer 2

I've tested this using different versions of the official Varnish Docker image. To be honest: I can't spot any behavioral differences between the different versions.

I used the varnish:latest, varnish:7.1 and even varnish:stable to perform the tests. In all the test scenarios only Vary: Cookie, User-Agent came out. I didn't spot the Accept-Encoding header in the Vary header on any of the versions.

Ensure your backend returns gzip-encoded responses

This led me to do some more research and turns out the standard nginx:alpine image I'm using for the test doesn't have gzip enabled in the vhost configuration.

This means that if your backend server doesn't return a Content-Encoding: gzip header, Varnish will not add a Vary: Accept-Encoding variation.

Setting gzip on; in my Nginx config didn't help. Apparently setting gzip_proxied any; did the trick, and all of the sudden I'm getting a Vary: Cookie, User-Agent, Accept-Encoding header.

I'm not saying my solution will work for you, but it's worth checking if your backend is actually returning gzip-encoded content.

Troubleshooting

You can use the following command to debug this in Varnish:

sudo varnishlog -g request -b -i berequrl -I BerespHeader:Content-Encoding -I BerespHeader:Vary -i Gzip -q "ReqUrl eq '/'"

Be sure to run this on a empty cache in order to catch the backend fetch. This example also assumes we're testing this on the home page

The output could look like this for gzip-encoded content:

**  << BeReq    >> 65541
--  BereqURL       /
--  BerespHeader   Vary: Cookie, User-Agent
--  BerespHeader   Content-Encoding: gzip
--  BerespHeader   Vary: Cookie, User-Agent, Accept-Encoding
--  Gzip           u F - 37 11 80 216 226

For plain text output, this is what the output could look like:

**  << BeReq    >> 3
--  BereqURL       /
--  BerespHeader   Vary: Cookie, User-Agent

Conclusion

There don't seem to be any behavioral changes in Varnish regarding GZIP compression and cache variations.

Varnish will only add Vary: Accept-Encoding if the backend actually returns gzip-encoded content. Web server configurations prevent gzip-encoded content for proxied requests.

Please use the troubleshooting tips mentioned above to examine this.