CODEWITHSUNDEEP
javascripthtmlseo
ariel
ariel

Reputation: 3072

Can I create a script to see which bots are crawling the site?

I have a bunch of clients for whom I want to customize the site based on what bot is crawling their site. Is it possible to create a script that detects which search engine bot is crawling the site and then track that or take another action?

Upvotes: 1

Views: 168

Answers (2)

ianaya89
ianaya89

Reputation: 4233

You can't do that because JavaScript (files or snippets) is not crawled by the bots. Only your text and you are html are crawled. Basically all the content served by your "server side" (your HTTP Response).

Upvotes: 0

MinusFour
MinusFour

Reputation: 14423

Many bots got an specific User Agent. You could filter out the bots with it and then just do whatever action you want to do. Not sure if you can do it from frontend (the javascript tag is present here, unless we are talking about node.js).

For an instance googlebot user agent:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

From PHP:

if($_SERVER['HTTP_USER_AGENT'] == 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)') { 
  doSomething();
}

The reason why I think you can't do it from frontend js is because many bots don't actually use javascript, they just read the http responses.

Edit: There are some bots that do interpret Javascript (googlebot is one of them), although to a certain extent.

Upvotes: 1

Related Questions