CODEWITHSUNDEEP
web-scrapingscrapy
Jason Youk
Jason Youk

Reputation: 802

Scrapy stats Crawled pages and pages/min

Could someone help me understand scrapy stats. I'm trying to optimize the scrapy crawling speed for an aws instance. My current scrapy crawl speed is INFO: Crawled 32429 pages (at 72 pages/min), scraped 197 items (at 0 items/min)
If scrapy is crawling at 72 pages/min what is 32429 pages? Definitely not 32429 pages/sec...

Upvotes: 1

Views: 374

Answers (1)

alecxe
alecxe

Reputation: 473803

Crawled X pages it is total amount of HTTP responses that Scrapy received while crawling.

FYI, logstats module is responsible for the report that you see on the console.

Upvotes: 2

Related Questions