CODEWITHSUNDEEP
wordpressgoogle-searchrobots.txt
Kevin.a
Kevin.a

Reputation: 4296

Do not index certain pages on Google Search

Is it possible to not index certain pages on Google Search?

I'm using WordPress. I've heard a robots.txt can prevent bots from crawling your website, but is this the same as indexing? Is it even possible at all?

Upvotes: 0

Views: 225

Answers (1)

Ben
Ben

Reputation: 5129

The Web Robots Pages

Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.

To exclude all robots from the entire server

User-agent: *
Disallow: /

To exclude all robots from part of the server

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

Upvotes: 2

Related Questions