Reputation: 799
Recently I find Cloudflare will add X-Robots-Tag(noindex, follow) to some of our pages. For example:
https://www.datanumen.com/fr/blogs/chatgpt-join-drawing-exam/
The response header in Chrome is below:
To make sure the flag is set by Cloudflare, I do the following check:
setDns www.datanumen.com ##.##.##.##
navigate https://www.datanumen.com/fr/blogs/chatgpt-join-drawing-exam/
Then the response header will NOT contain X-Robots-Tag any more, the result is at https://www.webpagetest.org/result/231201_AiDcS6_52V/1/details/#waterfall_view_step1
So how to prevent Cloudflare from adding X-Robots-Tag?
I check some related posts:
https://community.cloudflare.com/t/apo-cache-inserts-x-robots-tag-noindex-follow-header/377185 https://community.cloudflare.com/t/x-robots-tag-setting/58774 https://community.cloudflare.com/t/noindex-sent-to-google-crawling-bot/51113/10
but still cannot solve the issue.
Upvotes: 0
Views: 282