CODEWITHSUNDEEP
robots.txt
Henry
Henry

Reputation: 3013

Robots.TXT Disallow Syntax

I have dynamically created pages, which can have any name. 

www.example.com/pages/ 
www.example.com/pages/dynamicpage1/ 
www.example.com/pages/dynamicpage2/

I want the first url to pages to be crawled, but not any of the dynamic pages

User-agent: *
Disallow: /pages/*

The above will block all three pages listed. How do I just block all the dynamic pages, but not the first url.

Upvotes: 3

Views: 133

Answers (1)

Henry
Henry

Reputation: 3013

I figured it out.. 

User-agent: *
Disallow: /pages/*/

Happens all the time, solve the problem seconds after you post it.

Upvotes: 4

Related Questions