CODEWITHSUNDEEP
wordpresssearch-enginesitemaprobots.txtgoogle-search-console
Rocco The Taco
Rocco The Taco

Reputation: 3777

Sitemap contains urls which are blocked by robots.txt

We've had an accidental situation where our Wordpress site had the robots.txt set to disallow crawling for about 7 days. I'm now trying to play clean up and Webmaster Tools is saying "Sitemap contains urls which are blocked by robots.txt." AFTER I"ve adjusted the robots.txt and allowed crawling. There is no reason why the URLS would still be blocked and when I visit the examples they appear OK.

Robots.txt URL: http://bit.ly/1u2Qlbx

Sitemap URL: http://bit.ly/1BfkSmx

My URLs where the robots.txt is blocking according to Webmaster Tools: http://bit.ly/1uLBRea OR http://bit.ly/1CsrHnr

Upvotes: 5

Views: 10079

Answers (3)

Jérôme Verstrynge
Jérôme Verstrynge

Reputation: 59576

Don't worry, if you enabled the URLs again, Google's web crawlers will notice and crawl your pages again. The message will disappear from GWT within days.

Eventually, you may test your robots.txt with GWT's corresponding feature.

Upvotes: 6

hashtriplezero
hashtriplezero

Reputation: 81

Google can take a while to re-crawl your site. I'd say waiting is probably your only option.

I've had it take up to 7 days to index things properly after submitting a complete site map via webmaster tools.

It looks like you're using Yoast SEO, that plugin should tell you if any other issues exist.

Upvotes: 4

sprytechies
sprytechies

Reputation: 465

use this plugin

https://wordpress.org/plugins/wp-robots-txt/

it will remove previous robots.txt and set simple wordpress robots.txt and wait for a day

problem can be solved.

Upvotes: 5

Related Questions