michele
michele

Reputation: 26598

Allow crawling of only the home page of a sub-directory using robots.txt

I have

and

I would allow crawling for the entire example.com and only the home page of example.com/sitetwo. What I have to write in my robots.txt?

Upvotes: 0

Views: 95

Answers (1)

Mike
Mike

Reputation: 1999

Allow: /sitetwo/$
Disallow: /sitetwo/

$ means "exact match" in this context.

Upvotes: 1

Related Questions