CODEWITHSUNDEEP
robots.txtgooglebot
user1227914
user1227914

Reputation: 3514

How to deny Googlebot only for a specific set of page variables?

I have a page that is https://www.somedomain.com and then under that page I have the option for users to change the language, like

https://www.somedomain.com/?change_language=en&random_id=123
https://www.somedomain.com/?change_language=de&random_id=123
https://www.somedomain.com/?change_language=fr&random_id=123

etc.

Is it possible to deny Googlebot from crawling these links, but still crawl the https://www.somedomain.com/ main page?

Upvotes: 1

Views: 67

Answers (1)

eywu
eywu

Reputation: 2724

You can use robots.txt to target just the query parameter:

User-agent: *
Disallow: /?change_language

This will prevent Google or other good bots from crawling the language options on the homepage. If you want to make it more universal to all pages:

User-agent: *
Disallow: ?change_language

However, you might want to consider letting those language changes to be crawled and instead utilize the rel="alternate" hreflang specification that Google and Bing support.

This way you can indiciate to the engines that the content is in multiple languages allowing your site to get indexed in all the different country specific versions of Google, Bing, and Yahoo.

Upvotes: 1

Related Questions