CODEWITHSUNDEEP
restrictiongooglebot
pang
pang

Reputation: 4154

Google Crawler Time Restriction

does anyone know that it is possible to setup any property in order to inform googlebot to just come and crawl the site during specific day or time period (eg. during the weekend only)?

thanks,

Upvotes: 0

Views: 1098

Answers (3)

Michael Borgwardt
Michael Borgwardt

Reputation: 346317

You can use an XML sitemap to give a hint about the appropriate crawling frequency, but it's only a hint, and requesting specific days is not possible.

Upvotes: 2

mas
mas

Reputation: 1117

You can advise googlebots that you prefer a slower crawl rate (if your site is being crawled at faster than the lowest rate) but this takes effect for 90 days (see http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=48620).

Changing the robots.txt could be problematic as that is cached by Google, so disallowing crawling could result in the site not being crawled for far longer than intended.

Google has more than one bot type so you may be able to be selective over which parts of the site are appropriate for each of them to crawl, using robots.txt as it is intended. See http://www.google.com/support/webmasters/bin/answer.py?answer=40360.

Upvotes: 1

Jeffrey Kemp
Jeffrey Kemp

Reputation: 60262

Don't think so, but googlebot does re-read your robots.txt quite frequently, so I wonder if it would work if you swapped in an alternative robots.txt at those times, e.g. with a script?

Upvotes: 0

Related Questions