CODEWITHSUNDEEP
php-curlbing
MandyShaw
MandyShaw

Reputation: 1146

indexnow submissions: queries on 200 status codes and on how this is reflected in Bing Webmaster Tools

Having found out that only about 20% of my website pages (and not including the home page) are indexed by Bing (where Google indexes them all happily), I have been trying the IndexNow functionality recommended by Bing Webmaster Tools.

I have successfully submitted my 4,500 odd URLs using PHP (from the Windows command line, PHP code below).

However:

a) I seem to get a 200 status code all the time, even if my submitted JSON is definitely incorrect (bad key, for example). Is this normal? (I do get a 301 if I switch redirects off, so I don't think it's my code that's at fault here.)

b) The Bing Webmaster Tools IndexNow status page shows what appears to be a correct count of submissions, but only lists a small subset of the URLs I have submitted (it looks to be one per 5 minutes). Is this normal behaviour?

c) I am getting nothing at all on the response beyond the 200 status code (I temporarily tweaked the code to look for it). Given b) above, how can I check that each URL has gone through correctly and that there is no further action I need to take until the next time I make changes or additions to my website?

My PHP code:

function indexNow($param) {

// initiate the curl request
$request = curl_init();

$data = array('host' => "<myhostname>",
'key' => "<mykey>",
'keyLocation' => "https://<myhostname>/<mykey>.txt",
'urlList' => ["<myURLstem>".$param]);

curl_setopt($request, CURLOPT_URL,"https://api.indexnow.org/indexnow");
curl_setopt($request, CURLOPT_HTTPHEADER, array('Content-Type:application/json; charset=utf-8'));
curl_setopt($request, CURLOPT_POST, 1);
curl_setopt($request, CURLOPT_POSTFIELDS, json_encode($data, JSON_UNESCAPED_UNICODE)); 

// handle redirect and catch the response
curl_setopt($request, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($request, CURLOPT_RETURNTRANSFER, true);

$response = curl_exec($request);

if ($response === false) {
  echo('Error occurred: ' . curl_error($request));
}
echo('Status code for '.$param.': ' . curl_getinfo($request, CURLINFO_HTTP_CODE));

curl_close ($request);
}

Upvotes: 0

Views: 39

Answers (1)

Raghavendra N
Raghavendra N

Reputation: 3082

Answers to your questions:

I seem to get a 200 status code all the time, even if my submitted JSON is definitely incorrect (bad key, for example). Is this normal?

I did try your script with several invalid key and data combinations and I got these response codes: 202, 422 and 400. You can see the full list of response codes here.

So if you are getting 200 response code it indicates that your URL/set of URLs are submitted to the search engine successfully.

If search engines like your URLs and have enough crawl quota for your site, search engines will attempt crawling some or all these URLs.

The Bing Webmaster Tools IndexNow status page shows what appears to be a correct count of submissions, but only lists a small subset of the URLs I have submitted. Is this normal behaviour?

The HTTP 200 response code only indicates that the search engine has received your set of URLs. Search engines will crawl URL's according to their scheduling logic. And sometimes they can choose not to crawl and index URLs if they do not meet their selection criterion.

Using IndexNow ensures that search engines are aware of your website changes but it does not guarantee that web pages will be crawled or indexed by search engines. It may take time for the change to reflect in search engines.

So, yes in your case, this is expected behaviour.

I am getting nothing at all on the response beyond the 200 status code. How can I check that each URL has gone through correctly and that there is no further action I need to take?

At this point, if you are getting 200 status code, there is nothing much you can do. It is upto the Search engines to crawl and index your submitted URL's.

But you can always use sitemaps and keep it updated. Search engines will visit sitemaps every few days.

Upvotes: 0

Related Questions