CODEWITHSUNDEEP
robots.txt
Pa3k.m
Pa3k.m

Reputation: 1156

Does robots.txt apply to subdomains?

Let's say I have a test folder (test.domain.com) and I don't want the search engines to crawl in it, do I need to have a robots.txt in the test folder or can I just place a robots.txt in the root, then just disallow the test folder?

Upvotes: 28

Views: 20725

Answers (3)

malexander
malexander

Reputation: 4680

Each subdomain is generally treated as a separate site and requires their own robots.txt file.

Upvotes: 40

randomusername
randomusername

Reputation: 8097

When the crawler fetches test.domain.com/robots.txt that is the robots.txt file that it will see. It will not see any other robots.txt file.

Upvotes: 4

Kleskowy
Kleskowy

Reputation: 2668

If your test folder is configured as a virtual host, you need robots.txt in your test folder as well. (This is the most common usage). But if you move your web traffic from subdomain via .htaccess file, you could modify it to always use robots.txt from the root of your main domain.

Anyway - from my experience it's better to be safe than sorry and put (especially declining access) files robots.txt in all domains you need to protect. And double-check if you're getting the right file when accessing:

http://yourrootdomain.com/robots.txt
http://subdomain.yourrootdomain.com/robots.txt

Upvotes: 1

Related Questions