CODEWITHSUNDEEP
authenticationrobots.txt
Martti Laine
Martti Laine

Reputation: 12985

Correct way to add robots.txt and hide it?

I have a secret folder on my hosting, which may not be seen by visitors. I've added a robots.txt to htdocs:

User-agent: *
Disallow: /super-private/

However, if a visitor goes to http://example.com/robots.txt, he can see the name of the private folder. Is there anything to be done? Htaccess maybe?

Upvotes: 3

Views: 5015

Answers (2)

Will Vousden
Will Vousden

Reputation: 33358

robots.txt is not the solution here. All it does is tell things like search engine spiders that a particular URL should not be indexed; it doesn't prevent access.

Put a .htaccess file in super-private containing the following:

Deny From All

Once you've done this, there's no need for robots.txt, as it'll be inaccessible anyway. If you want to allow access to certain people, then look into authentication with .htaccess.

Upvotes: 8

Darin Dimitrov
Darin Dimitrov

Reputation: 1038770

Don't mention this private folder in robots.txt. Then simply disallow the access to it with .htaccess:

deny from all

Also if there are no links to this super-private folder in the other pages robots should never know if its existence but disallowing the access is a good thing to do if this folder should never be directly accessed from clients.

Upvotes: 2

Related Questions