CODEWITHSUNDEEP
.htaccessrobots.txt
Corporate Film Makers Mumbai
Corporate Film Makers Mumbai

Reputation: 31

301 redirects to all except robots.txt

We have moved our website to a new domain and want all pages of the old site to get removed from search engines. It's the same site, same content, just a new domain, so search engines are taking time because of duplicate content (maybe). We have added .htaccess 301 from our old site to new site as:

redirect 301 / http://new-domain.com/

Now, to remove our old site from search engines, we changed our robots.txt on the old site as:

User-agent: *
Disallow: /

The problem is, the search engines are fetching robots.txt from the new-domain.com because of .htaccess 301 redirect.

How do I restrict 301 redirect for robots.txt?

Upvotes: 3

Views: 4507

Answers (2)

Shashikant Nishant Sharma
Shashikant Nishant Sharma

Reputation: 11

I was trying different configurations and spent hours and finally I found this code which is working well and hope to get the desired results Here is the code for disallowing search engines. 

User-agent: *
Disallow: /

Upvotes: -3

Gumbo
Gumbo

Reputation: 655219

Remove the Redirect directive and try this mod_rewrite rule:

RewriteEngine on
RewriteRule !^robots\.txt$ http://other.example.com%{REQUEST_URI} [L,R=301]

This will redirect any request except /robots.txt.

Upvotes: 7

Related Questions