CODEWITHSUNDEEP
seouser-agentweb-crawler
Himberjack
Himberjack

Reputation: 5792

How to know if HTTP Request is a BOT

I am looking for a full list of User-Agents of BOTS (crawlers, spiders, twitter bots, etc).

Do you know anything?

Thanks

Upvotes: 8

Views: 9750

Answers (2)

Ben
Ben

Reputation: 16553

Long story short: you can't, there's no silver bullet. Any bot could set their user-agent string to anything from 'googlebot' to 'spamalot'.

You can see it yourself, all you need to do is go to the first site Shinnok pointed, and start counting all those Googlebot/2.X bots listed in there. You block them, they change the name of the bot to random gibberish and so on. In the end you'll end up with a 10k bots list that will decrease your users load times when you try to verify if they're a bot or not.

Upvotes: 3

Shinnok
Shinnok

Reputation: 6389

Check this list: http://www.botsvsbrowsers.com/category/1/index.html

It contains a total number of 4768 bot user agents.

The other way around to accomplishing bot detection is to use the reverse approach in a white-list way, that is, check if the user agent is not a bot, then anything else is a bot. :-)

To compile a comprehensive list of non bot user agents you can use the lists at http://www.user-agents.org/ and http://www.botsvsbrowsers.com/.

Upvotes: 3

Related Questions