CODEWITHSUNDEEP
web-crawler
Angelo
Angelo

Reputation: 1

where I to put file robots.txt?

I have the domain www.mydomain.com and I set apache mod-rewrite so as to have www.mydomain.com/myappl. Where should I place the file robots.txt? thanks!

Upvotes: 0

Views: 258

Answers (2)

paradigmatic
paradigmatic

Reputation: 40461

I don't know ;-) But you can use the robots checker to find if your file is accessible. So if you don't know guess where it will be accessible and test it.

Upvotes: 0

sleske
sleske

Reputation: 83635

The robots.txt must be accessible to clients as http://www.mydomain.com/robots.txt . How to accomplish that depends on your mod-rewrite configuration. Please post the config if you need more help.

Also see http://en.wikipedia.org/wiki/Robots_exclusion_standard for details.

Upvotes: 2

Related Questions