CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Thibault Randria
Thibault Randria

Reputation: 95

Scrapy - Timing to get a Response

Just like in Scrapy request+response+download time, I would like to know how many time it takes to get a Response. The solution proposed doesn't meet my needs because of the following issue:

When a Downloader Middleware process_request method returns a Request object, the Request is rescheduled and isn't passed immediately to the remaining process_request methods. As a consequence, the solution proposed will include the time needed for the scheduler to return the Request to the Engine again.

What I want is only the time the Downloader takes to download a page (the time elapsed between the end of Downloader Middleware processings of the Request and first Downloader Middleware processing of the Response).

My idea is that one could either:

  1. Disable the rescheduling of returned Request. But is it desirable and how can we do this?
  2. Or Try to use the 'timer' used to trigger a TimeoutError. But I don't know how to access it.

Thanks in advance!

Upvotes: 3

Views: 913

Answers (1)

Tomáš Linhart
Tomáš Linhart

Reputation: 10210

Isn't this exactly what download_latency request meta key contains? Or your requirement is different?

Upvotes: 3

Related Questions