Reputation: 57
I'd like to know if there's any method to view headers that are being sent when scrapy issues a POST/GET request within live logs, shell or by any other similar means. Thanks!
Upvotes: 6
Views: 10369
Reputation: 4501
Both Response
and Request
objects will have their headers available via the .headers
attribute.
Headers for both objects are modified via the Middleware between the Downloader and the Engine (see Scrapy Architecture). If you create a new Request
object, it won't have any headers until it's passed through the Middleware that assigns them.
To view the request object, as it will be sent out, you will need to create a Middleware, put it closer to the Downloader than any other header-altering Middleware, and check the request.headers
attribute at that time.
Alternatively, you can check out the headers of the request which returned a Response
to the Spider by viewing the response.request.headers
. This may not be the Request
object you sent out though, but the one that the resulted in the Response
object that was returned (for instance, redirects/retries result in the originally dispatched Request
object being different than the Request
object in response.request
). Of course, this requires a Response
object to have been returned to the spider, so it won't work for any Request
that didn't generate a response (eg. DNS lookup error), or any Response
that gets ignored or dropped via Middleware (eg. HTTP Status 400).
Upvotes: 14
Reputation: 18799
when you make a request, you catch it on a callback method, the callback should be like:
def parse_method(self, response):
there you can check the response headers with response.headers
Upvotes: -2