CODEWITHSUNDEEP
python-2.7scrapyscrapydportia
Karthick
Karthick

Reputation: 55

How do i get request and response count in scrapyd?

I am trying to get request and response count in scrapyd,while running multiple spider means 8 spider dynamically.I am try to get those count using python.

following counts:

enter code here{'downloader/request_bytes': 130427,
 'downloader/request_count': 273,
 'downloader/request_method_count/GET': 273,
 'downloader/response_bytes': 2169984,
 'downloader/response_count': 273,
 'downloader/response_status_count/200': 271,
 'downloader/response_status_count/404': 2,
 'dupefilter/filtered': 416,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2015, 7, 21, 14, 21, 38, 13000),
 'item_scraped_count': 119,
 'log_count/DEBUG': 406,
 'log_count/INFO': 44,
 'offsite/domains': 9,
 'offsite/filtered': 5865,
 'request_depth_max': 14,
 'response_received_count': 273,
 'scheduler/dequeued': 273,
 'scheduler/dequeued/memory': 273,
 'scheduler/enqueued': 273,
 'scheduler/enqueued/memory': 273,
 'start_time': datetime.datetime(2015, 7, 21, 14, 16, 41, 144000)}
enter code here

Thanks,

Upvotes: 0

Views: 436

Answers (1)

GHajba
GHajba

Reputation: 3691

Use the Stats Collection from Scrapy.

With this you can access the statistics dumped at the end to the console and if you write your own middleware you can combine the results of your 8 spiders together -- like in the example of the documentation.

Upvotes: 1

Related Questions