CODEWITHSUNDEEP
robots.txt
Zerium
Zerium

Reputation: 17333

How to block all URLs in robots.txt except for the directory indexes?

For example, I want to block /foo.php, /foo/foo.php and every other similar URL in robots.txt, only leaving /, /foo/, etc. behind.

In other words, I want to block everything except the directories.

How is this possible?

Upvotes: 0

Views: 145

Answers (1)

unor
unor

Reputation: 96567

If all URLs you want to block end in (or contain) .php, you could use

User-agent: *
Disallow: /*.php

However, this uses the * wildcard, which maybe not all parsers support (Google supports it).

Upvotes: 1

Related Questions