CODEWITHSUNDEEP
seorobots.txt
Learner
Learner

Reputation: 786

robots.txt for disallowing Google to not to follow specific URLs

I have included robots.txt in the root directory of my application in order to tell Google bots that do not follow this http://www.test.com/example.aspx?id=x&date=10/12/2014 URL or the URL with the same extension but different query string values. For that I have used following piece of code:

User-agent: *
Disallow: 
Disallow: /example.aspx/

But I found in the Webmaster Tools that Google is still following this page and has chached a number of URLs with the specified extension, is it something that query strings are creating problem because as far as I know that Google do not bother about query string, but just in case. Am I using it correctly or something else also needs to be done in order to achieve the task.

Upvotes: 0

Views: 125

Answers (1)

Ciseur
Ciseur

Reputation: 463

Your instruction is wrong :

Disallow: /example.aspx/

This is blocking all URLs in the direcory /example.aspx/

If you want to block all URLs of the file /example.aspx, use this instruction:

Disallow: /example.aspx

You can test it with Google Webmaster Tools.

Upvotes: 1

Related Questions