CODEWITHSUNDEEP
seoweb-crawlerrobots.txt
todd
todd

Reputation: 101

How to allow crawlers access to index.php only, using robots.txt?

If i want to only allow crawlers to access index.php, will this work?

User-agent: *
Disallow: /
Allow: /index.php

Upvotes: 10

Views: 16761

Answers (5)

mRGogo
mRGogo

Reputation: 21

User-agent: *
Allow: /$
Allow: /index.php
Allow: /sitemap.xml
Allow: /robots.txt
Disallow: /

Sitemap: http://www.your-site-name.com/sitemap.xml

Upvotes: 2

bulava
bulava

Reputation: 31

User-agent: *

Allow: /index.php
Disallow: /

Upvotes: 3

Simone Carletti
Simone Carletti

Reputation: 176472

Yes, it will work. Here's the test result from the Google Webmaster Tool.

Url
http://www.example.org/index.php

Googlebot
Allowed by line 3: Allow: /index.php

Googlebot-Mobile
Allowed by line 3: Allow: /index.php

However, remember that with this configuration your site homepage won't be crawled unless the page is accessed with the full qualified path. In other words, http://www.example.org/ is forbidden while http://www.example.org/index.php is allowed.

If you want your homepage to be accessible, here's a better version of your file.

User-agent: *
Disallow: /
Allow: /index.php
Allow: /$

Upvotes: 22

UpTheCreek
UpTheCreek

Reputation: 32391

Try swapping the order of Disallow / Allow:

User-agent: *
Allow: /index.php
Disallow: /

See this info from wikipedia:

"Yet, in order to be compatible to all robots, if you want to allow single files inside an otherwise disallowed directory, you need to place the Allow directive(s) first, followed by the Disallow, for example:"

http://en.wikipedia.org/wiki/Robots.txt

Still I wouldn't expect it to work too consistently

Upvotes: 3

Janco
Janco

Reputation: 1140

You can use the Google Robots tool to checkout. I would never put any secret directories in the robots file as I would guess that a line like below would be as honey for certain spiders.

Disallow: /secret

Upvotes: 2

Related Questions