CODEWITHSUNDEEP
http
bvs
bvs

Reputation: 1087

Are all spiders supposed to use +http in their user agent string?

Here are some spider user agent strings I've seen recently. They all seem to include a URL prefixed with +:

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Is that just a convention that most spiders follow? Or is it specified somewhere. I couldn't find it.

Upvotes: 2

Views: 221

Answers (1)

David
David

Reputation: 34573

It's just a convention that some spiders follow. There is no constraint on what people can put in a user agent header.

Take a look at this list of user agents that contain "GoogleBot". You'll notice that many of these don't contain "+http".

Upvotes: 2

Related Questions