CODEWITHSUNDEEP
pythonrequestscrapyweb-crawler
HospiCZ
HospiCZ

Reputation: 57

Scrapy: repeating request for 440 status code

Is there a way in scrapy how to repeat a request, if the response status code is 440?

Sometimes the server i am trying to scrape responds with 440 code and the I can´t scrape the data.

Thanks a lot!

Upvotes: 1

Views: 187

Answers (1)

dethos
dethos

Reputation: 3454

Based on the scrapy documentation you can add and configure the RetryMiddleware to behave the way you want.

The settings you will want to change are: RETRY_TIMES and RETRY_HTTP_CODES. Specially the second one where the defaults are: [500, 502, 503, 504, 408], so there you just need to add 440 to the list.

Check more details about the middleware here

Upvotes: 1

Related Questions