CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Spaceman
Spaceman

Reputation: 1205

Scrapy: catch responses with specific HTTP server codes

We have a pretty much standard Scrapy project (Scrapy 0.24).

I'd like to catch specific HTTP response codes, such as 200, 500, 502, 503, 504 etc.

Something like that:

class Spider(...):

    def parse(...):
        processes HTTP 200

    def parse_500(...):
        processes HTTP 500 errors

    def parse_502(...):
        processes HTTP 502 errors

    ...

How can we do that?

Upvotes: 10

Views: 6810

Answers (1)

alecxe
alecxe

Reputation: 474001

By default, Scrapy only handles responses with status codes 200-300.

Let Scrapy handle 500 and 502:

class Spider(...):
    handle_httpstatus_list = [500, 502]

Then, in the parse() callback, check response.status:

def parse(response):
    if response.status == 500:
        # logic here
    elif response.status == 502:
        # logic here

Upvotes: 14

Related Questions