CODEWITHSUNDEEP
robots.txt
radiofrequency
radiofrequency

Reputation: 873

What do these lines in the Google+ robots.txt mean?

http://plus.google.com/robots.txt has the following contents:

User-agent: *
Disallow: /_/

I'm assuming this means search engines are allowed to index anything in the first level off the root and nothing further?

Upvotes: 3

Views: 227

Answers (1)

nandhp
nandhp

Reputation: 4817

I believe those lines are used to deny robots access to URLs like https://plus.google.com/_/apps-static/_/ss/landing/... and https://plus.google.com/_/apps-static/_/js/landing/....

These URLs mainly appear to be CSS, Javascript, and JSON, but there might be other things (that are more valuable to search engines) which aren't immediately obvious.

Upvotes: 4

Related Questions