CODEWITHSUNDEEP
indexingseometagoogle-console-developer
mauxtin
mauxtin

Reputation: 408

Prevent search engines from indexing my api

I have my api at api.website.com which requires no authentication.

I am looking for a way to disallow google from indexing my api.

Is there a way to do so?

I already have the disallow in my robots at api.website.com/robots.txt but that just prevents google from crawling it.

User-agent: *
Disallow: /

The usual way would be to remove the Disallow and add a noindex meta tag but it's an API hence no meta tags or anything.

Is there any other way to do that?

Upvotes: 2

Views: 4560

Answers (2)

mauxtin
mauxtin

Reputation: 408

It seems like there is a way to add a noindex on api calls.

See here https://webmasters.stackexchange.com/questions/24569/why-do-google-search-results-include-pages-disallowed-in-robots-txt/24571#24571

The solution recommended on both of those pages is to add a noindex meta tag to the pages you don't want indexed. (The X-Robots-Tag HTTP header should also work for non-HTML pages. I'm not sure if it works on redirects, though.) Paradoxically, this means that you have to allow Googlebot to crawl those pages (either by removing them from robots.txt entirely, or by adding a separate, more permissive set of rules for Googlebot), since otherwise it can't see the meta tag in the first place.

Upvotes: 3

Matt Makris
Matt Makris

Reputation: 52

It is strange Google is ignoring your /robots.txt file. Try dropping an index.html file in the root web directory and adding the following between the <head>...</head> tags of the web page.

<meta name="robots" content="noindex, nofollow">

Upvotes: 0

Related Questions