CODEWITHSUNDEEP
search-enginerobots.txt
Leo Leoncio
Leo Leoncio

Reputation: 1709

Disallow group of folders using robots.txt

I need to disallow the access without inform at robots.txt which folder is that.

In my case I have three folders /_private1, /_private2 and /_private3.

I need to know if I use like above, I'm going to protect my folder against Google and others. How should I do that?

Disallow: /_*
Disallow: /_

Upvotes: 1

Views: 175

Answers (1)

plasticinsect
plasticinsect

Reputation: 1752

To disallow any directory or file whose name begins with an underscore:

User-agent: *
Disallow: /_

You should be aware that this does not completely guarantee that these directories will never show up on any search engines. It prevents robots.txt compliant robots (which includes all major search engines) from crawling them, but they could still theoretically show up in a search if someone decides to link to them. If you really need to keep this info private, best practice is to put it behind a password.

Upvotes: 1

Related Questions