CODEWITHSUNDEEP
.htaccessrobots.txt
Bill Masters
Bill Masters

Reputation: 161

.htaccess not letting robot.txt through

I have the following .htaccess file in my root:

Options +FollowSymlinks
RewriteEngine on
RewriteBase /
RewriteRule ^(.*)\/(\?.*)?$ $1$2 [R=301,L]
RewriteRule ^([^/]*)$ index.php?page=$1 [NC]

This works as it should for shortening all my URLs to website.com/something

The problem is Google can't find my robots.txt file in my root. The above file isn't letting it through. when It type website.com/robots.txt I get a 404 not found. But if I comment out the above .htaccess code I can get to it just fine.

How can I edit my .htaccess file to let robots.txt through without interfering with my other URLs?

Upvotes: 4

Views: 17596

Answers (4)

Kevern
Kevern

Reputation: 1

Find the line that already exists in your .htaccess that says this:

RewriteRule ^sitemap.xml$ index.php?route=feed/google_sitemap [L]

And change it to this:

RewriteRule ^itemap.xml$ index.php?route=feed/google_sitemap [L]

Upvotes: 0

Bill Masters
Bill Masters

Reputation: 161

I tried both suggestions and they both work great. However I went with Kiran's answer simply because it's a shorter syntax. This is what I ended up with.

Options +FollowSymlinks 
RewriteEngine on

RewriteBase /

# remove trailing slash
RewriteRule ^(.*)\/(\?.*)?$ $1$2 [R=301,L]

# Allow Robots.txt to pass through
RewriteRule ^robots.txt - [L]

RewriteRule ^([^/]*)$ index.php?page=$1 [NC]

Upvotes: 2

linuxeasy
linuxeasy

Reputation: 6499

You can use this solution in your .htaccess file:

Options +FollowSymlinks
RewriteEngine on
RewriteBase /
RewriteCond $1 !^(index\.php|images|robots\.txt)
RewriteRule ^(.*)$ index.php?page=$1 [L]

This rewrites all your requests to index.php?page=, except the files specified in the RewriteCond list.

Upvotes: 1

Kiran
Kiran

Reputation: 921

RewriteEngine on
RewriteRule ^robots.txt - [L]

Second line will exclude robots.txt from URL rewritting rules . Try above code

Upvotes: 13

Related Questions