CODEWITHSUNDEEP
javascriptseogoogle-search-console
clemlatz
clemlatz

Reputation: 8063

How to prevent Google from crawling partial url found in javascript?

I have recently added a site to Google Search Console and a lot of 404 errors are showing, all with the same pattern :

https://example.com/metas?some_id=247

That page does not exists and is linked to anywhere on the site. But I when check the source code fo the page that is to be the referer, I have a script tag with this code :

$.ajax({
  url: "/products/" + productId + "/metas?some_id=247",
  dataType: 'script',
  success: function(data) {
    // ...
  },
  error: function(XMLHttpRequest, textStatus, errorThrown) {
    // ...
  }
});

Why would Google bot crawl this url and how I can I prevent this from happening (obviously rel="nofollow" is out of the question here)? Would it help to put the script in a external .js file instead instead of a <script> tag in the page html code?

Upvotes: 3

Views: 1592

Answers (1)

inkovic
inkovic

Reputation: 313

It is generally considered a good idea to apply a noindex tag on URLs that don't exist.

Depending on your platform, simply edit your 404 page template to include <meta name="robots" content="noindex, nofollow" /> to solve a lot of issues.

For tricky automatically generated URLs due to a script, plugin or whatever you can use a URL pattern match to apply your noindex tag.

Here is an example in PHP:

$url = 'http://' . $_SERVER['SERVER_NAME'] . $_SERVER['REQUEST_URI'];

if (strpos($url,'YOURQUERY') !== false) {
echo '<meta name="robots" content="noindex, nofollow" /> '
}

Upvotes: 2

Related Questions