CODEWITHSUNDEEP
wordpressrobots.txt
Mr. Boy
Mr. Boy

Reputation: 63778

Does robots.txt apply to files/directories only, or URLs too?

I can use robots.txt to stop a folder of images/html files getting indexed. But what about dynamic pages, e.g. preventing indexing of certain WordPress pages?

Upvotes: 0

Views: 75

Answers (1)

Pekka
Pekka

Reputation: 449605

The robots.txt syntax doesn't care about whether a page is dynamic or not: All that matters for it is the directory structure.

If you are using a permalink structure like

example.com/blog/year/month/slug

you should be able to exclude single pages like so:

user-agent: *
disallow: /blog/2011/09/this-is-a-test-entry

you could use Google's webmaster tools to verify whether that happens properly.

Remember that Wordpress stores static content like images and PDF documents in /wp-content - you can't block those this way unless you want to block all resources in that directory.

Upvotes: 1

Related Questions