Rahul Prasad
Rahul Prasad

Reputation: 8222

Why google stopped crawling after a limit

I have submitted sitemap 1-2 months back. Google started crawling very rapidly for a few days and indexed about 25 % of pages then it suddenly stopped and its not crawling any more.

I have created a Google Alert for newly crawled pages. I am getting alerts everyday that some pages has been crawled (only new pages).

Robots.txt has settings to Allow All

Here are the screenshots, as you can see in the beginning curve is steep but then it became stagnant. This graph has been stagnant for a month

Crawled WebPages

Even Images are not being scrolled

Upvotes: 2

Views: 783

Answers (1)

jabbink
jabbink

Reputation: 1291

The official answer to this question can be found here: https://support.google.com/webmasters/answer/34441

Personally I also experienced this once, and it could be traced back to (sometimes) slow webserver response times, after which Google decided that it didn't want to slow my site even further and therefore only crawled new data.

Also, maybe Google crawled everything, but sees your other 75% of the pages as being duplicates and ignores them from now on (you should be able to see this in your webmaster tools) => Can you check your access log to see if Google maybe crawled many more pages?

Upvotes: 2

Related Questions