CODEWITHSUNDEEP
ruby-on-railsscalabilitysitemap
VP.
VP.

Reputation: 5151

google sitemap in a different server

problem

I have an web application, where as it should be, is hitted the whole time by crawlers. Most of them follows our google sitemap to discover new urls. I would like to move my sitemap to a different server, with a redis backend to try to offload my web app. My sitemap is huge, sitemap index with N * 50.000 urls and even caching it, it hurts my web server affecting direct my application response time when it is already under heavy usage.

questions

Upvotes: 1

Views: 199

Answers (1)

Elad
Elad

Reputation: 3130

It's not entirely clear from your question - is your sitemap generated dynamically for every request that passes through the cache?

If so, then the first thing I'd do is make it as static as possible. Write it to files, and serve them statically. I assume that most of your links don't change and it's just new pages that need to be added. The way I solved this in the past was to keep all the old links in static files, add run a background process that periodically adds new links to new sitemap files and adds those new files to the sitemap index.

Another thing: consider storing those static files already gzipp'ed and serving them as-is to avoid having your web server gzipping the same huge files on the fly again and again.

Finally, if serving the static sitemap still creates a load on your server, consider hosting it on S3 or some similar service. Since it's just static files, you don't need another web server and S3 would be a much cheaper option compared.

None of this should affect your pagerank. Spreading your application across multiple servers is common practice to handle load.

Upvotes: 1

Related Questions