weexpectedTHIS
Reputation: 3376
Google Webmaster Tools claiming my robots.txt blocking almost all of my site
I have submitted a sitemap which has many thousands of URLs, but when I look at the webmaster tools it claims that 9800 of my URLS are blocked by my robots.txt file.
What am I supposed to do to convince it that nothing is being blocked?
Upvotes: -1
Views: 118
Answers (1)
John Mueller
Reputation: 1481
Sometimes this just means that the robots.txt file couldn't be reached (returned a 5xx server error, or Googlebot just didn't get a response). In those cases, Google will treat any URL they attempt to crawl as being disallowed by robots.txt. You can see that in the Crawl Errors section in Webmaster Tools (in the site-errors on top).
Upvotes: -1
Related Questions
- How can i fix "Googlebot can't access your site" issue?
- Why is googlebot blocking all my urls if the only disallow I selected in robots.txt was for iisbot?
- Magento - robots.txt may be blocking URLs on Google Webmaster Tools
- Why is Google crawling pages blocked by my robots.txt?
- Google Webmaster Tools is not accepting my sitemap
- Why is Google Webmaster Tools completely misreading my robots.txt file?
- GWT says robots.txt is blocking resources - mine or theirs?
- Google Webmaster Tools won't index my site
- google webmaster tools content keywords come from robots.txt
- Url blocked by robots.txt message in Google Webmaster