CODEWITHSUNDEEP
seorobots.txtgoogle-search-console
Imran Sh
Imran Sh

Reputation: 1774

Why Google robots.txt Tester has error and it's not valid

As you can see in below image Google WebMaster Tools robots.txt Tester tell me about 9 error but I don't know how to fix it and what is the problem? enter image description here

Please help me to figure out

Upvotes: 1

Views: 1317

Answers (2)

Amir Amirloo
Amir Amirloo

Reputation: 19

you can use webmaster panel tools https://www.google.com/webmasters/tools/robots-testing-tool and test your robots file then download it

it works ok.

enter image description here

Upvotes: 0

L Martin
L Martin

Reputation: 1190

That is a valid robots.txt - but you've got a UTF-8 BOM (\xef\xbb\xbf) at the beginning of the text file. That's why there's a red dot next to 'User' in the first line. This mark tells browsers and text editors to interpret the file as UTF-8 whereas the robots.txt is expected to use only ASCII characters.

Convert your text file to ASCII and the errors will go away. Or copy everything after the red dot and try pasting it in again.

I tested this on the live version, here's the result translated from byte form:

\xef\xbb\xbfUser-agent: *\r\nDisallow: /en/news/iranology/\r\nDisallow:
 /en/tours-services/tour-the-soul-of-iran\r\nDisallow: 
/en/tours-services/tour-a-whistle-stop-tour\r\nDisallow: /en/to

You can clearly see the BOM at the beginning. Browsers and text editors will ignore it but it may mess with a crawlers ability to parse the robots.txt. You can test the live version using this python script:

import urllib.request

text = urllib.request.urlopen('http://www.best-iran-trip.com/robots.txt')

print(repr(text.read()))

If you're able to install Notepad++, it actually has an encoding menu that lets you save it in any format.

Upvotes: 5

Related Questions