CODEWITHSUNDEEP
seo
yossi
yossi

Reputation: 1431

How to hide robot.txt from vistors?

I have seen sites hide the robot.txt file.

if you enter the name of the site as http://www.mysite.com/robot.txt you will not receive the robot.txt,

I also want to hide the file robot.txt from visitors how to do it?

Is there a connection of these lines

<meta name="ROBOTS" content="NOODP">
         <meta name="Slurp" content="NOYDIR">

So I do not understand the meaning of the code. Thank you!

Upvotes: 0

Views: 643

Answers (1)

MatthewSot
MatthewSot

Reputation: 3594

I'm not sure exactly what you're asking, but couldn't you do that with URL rewrites? You might be able to display the robots.txt file for visitors with the User Agent strings of a crawler (for instance, "Googlebot"), and then redirect to a 404 if it's a non-crawler UA

Upvotes: 1

Related Questions