CODEWITHSUNDEEP
securitysearch-enginerobots.txt
stuartdotnet
stuartdotnet

Reputation: 3034

How to hide website directory from search engines without Robots.txt?

We know we can stop search engines from indexing directories on our site using robots.txt.

But this of course has the disadvantage of actually publicising directories we don't want found to possible attackers.

Password protecting the directory using .htaccess or other means is obviously the best way to keep the directory private.

But what if, for reasons of convenience, we didn't want to add another layer of security to the directory and just wanted to add another level of obfuscation? To hide, for example, an admin login page.

Is there another way to "hide" the directory without broadcasting its location in a robots.txt file?

Upvotes: 3

Views: 1070

Answers (2)

SilverlightFox
SilverlightFox

Reputation: 33538

Here is what to do, please note as you haven't mentioned any particular technology I haven't included how to do it.

If you configure your web server to output the following meta tag in the directory listing HTML page, it will prevent your page from being indexed by compliant search engines.

<meta name="robots" content="noindex">

Adding this would probably require implementing a custom module within your web server that will override the default directory listing output page.

Upvotes: 1

user3886075
user3886075

Reputation: 1

Try using a random string. Something like http://website.com/some-random-string-here/file.html

The remember not use some-random-string-here in your robots.txt file or on any links.

Upvotes: 0

Related Questions