CODEWITHSUNDEEP
pythonrequestscrapy
superdee
superdee

Reputation: 697

How can I rerun a scrapy request object in Python?

I am using Scrapy, and in the response_is_ban function, I have two objects, a {Request} object, and a {Response} object.

from scrapy.http import Request

For some reason, the Scrapy Request is returning different results than a standard Curl.

Is there a way I can rerun the {Request} object?

Something like:

response = request.get_again()

The goal is to figure out exactly what is different between my Curl function and the Scrapy request object, and why they would be returning different results. Any suggestions would be welcomed.

Edit: Another question I have, where in the code does Scrapy actually execture the request? Is there a request.get() function somewhere I could put a debug point on?

Upvotes: 0

Views: 162

Answers (1)

Thiago Curvelo
Thiago Curvelo

Reputation: 3740

For debugging, you could use fetch from a scrapy.shell session: https://docs.scrapy.org/en/latest/topics/shell.html#invoking-the-shell-from-spiders-to-inspect-responses

Upvotes: 1

Related Questions