CODEWITHSUNDEEP
.netcssdeploymentsquishit
chilly
chilly

Reputation: 194

404 issue with static assets, CDN and multi server environment

We have an issue in our production environment, maybe someone else has ran into this problem and has a smart solution.

Prerequisites:

Deployment routine:

  1. Take server1 off the load balancer.
  2. Make deploy to server1.
  3. Warm it up.
  4. Include server 1 in the load balancer.
  5. Take server2 off the load balancer (...and repeat steps 2-4)

And here's where it fails:

Between step 4 and 5 above, we have both servers online for a short period of time. During this time server1 could be referencing to main_46eb48ac968.css and server2 could be referencing the main_987eba4687.css. This will cause troubles in the following scenario...

Use case:

  1. User visits the site and end up on newly deployed server 1.
  2. Browser will request the main_46eb48ac968.css from the CDN.
  3. The CDN requests that file from the loadbalancer since it's not in its cache.
  4. The load balancer sends the CDNs request to server2.
  5. Server2 returs a 404 page not found error since the new file is only on server 1.
  6. Site looks like crap!

An easy fix would of course be to run without the CDN during deploy, but since CDN url is prepended to the paths during startup of the application we would have to restart the application in production... :/

Ideas?

Upvotes: 1

Views: 918

Answers (2)

chilly
chilly

Reputation: 194

Answer to @AlexCuse

Yes, our CDN is using the site as origin.

It was actually me who suggested the hash-as-virtual-folder-solution :) We used that solution until we deployed to the multi front servers environment where we discovered this issue. It's a great way of cache busting files in environments where you do not have CDN and multiple front end servers. Unfortunately we have both.

If we use the hash-as-virtual-folder solution, the old file is served with the new URL in 50% of the cases the short amount of time where both servers are online, one server with old code and one with new code. We could indeed flush the CDN after deploy, but that won't help the clients who already downloaded the wrong file since it has a max-age of 365 days and wont be updated until the client hit ctrl+F5. And we can't control that. Crap. ;)

So, our decision so far is that it's better for a user to get a crappy site without CSS/JS during like one minute (if he is the first visitor and hits the "wrong" server) than caching the wrong file which won't be updated until next deploy.

We also made some changes to our code, now we can disable the CDN while servers are in production without recycling the app pool by deleting the key storing the CDN URL in the http cache. It's a little tedious since it involves logging in to the CMS, do the change, save, publish and then change back, but it's possible.

Thanks anyway, it's a great library!

Upvotes: 0

AlexCuse
AlexCuse

Reputation: 18306

Hmm so your CDN is using your site as the origin I take it?

You might want to review this issue. What this does is writes the hash as a folder in the generated path, you can then use IIS rewrite rules to remove that folder from the URL on the way in. This was done to give an option that provides the reliable cache-busting of the "hash in filename" option without the explosion of files. You could use this to ensure that something gets served - but in this scenario it would leave you with an old version of the file in your CDN's cache (so you'd need to invalidate everything after you finish deployment).

This will be available in the next release (0.9.3)

Upvotes: 0

Related Questions