CODEWITHSUNDEEP
php.htaccessbotsrobots.txt
zozo
zozo

Reputation: 8582

Disallow robots can be bypassed with htaccess?

I have a simple question. Let's say that I have this in robots.txt:

User-agent: *
Disallow: /

And something like this in .htaccess:

RewriteRule ^somepage/.*$ index.php?section=ubberpage&parameter=$0

And of course in index.php something like:

$imbaVar = $_GET['section']
// Some splits some whatever to get a specific page

include("pages/theImbaPage.html") // Or php or whatever

Will the robots be able to see what's in that html included by the script (site.com/somepage)? I mean... the URL points to an inaccessible place... (the /somepage is disallowed) but still it is redirected to a valid place (index.php).

Upvotes: 0

Views: 1240

Answers (2)

Madara's Ghost
Madara's Ghost

Reputation: 174957

Assuming the robots will respect the robots.txt, then it wouldn't be able to see any page in the site at all (you stated you used Disallow: /.

If the robots however do not respect your robots.txt file, then they would be able to see the content, as the redirection is made server side.

Upvotes: 1

genesis
genesis

Reputation: 50976

No. By disallowing robot access, robots aren't allowed to browse any pages on your site and they're following your rules

Upvotes: 1

Related Questions