CODEWITHSUNDEEP
pythonscrapy
Flasking
Flasking

Reputation: 101

HTTP status code is not handled or not allowed

I need crawl a site with cookie using Scrapy,but returns error

code here

class XueqiuSpider(scrapy.Spider):
    name = "xueqiu"
    start_urls = ["https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1"]
    delimiter = ','
    quotechar = '"'
    headers = ["symbol","date","open","high","low","close","volume"]

    def start_requests(self):
        for i,url in enumerate(self.start_urls):
            print(url)
            yield Request(url,cookies={'aliyungf_tc':'AQAAANiAQ3xQ/QAAZ0J2fRFnxcJufEzG'},callback=self.parse_item)

    def parse_item(self, response):
        print response

errors show

********Current UserAgent:Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11************
2017-03-02 18:56:02 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1>
Cookie: aliyungf_tc=AQAAANiAQ3xQ/QAAZ0J2fRFnxcJufEzG; aliyungf_tc=AQAAAM/c+1g5vAMAZ0J2fbusPyBy7jb1

2017-03-02 18:56:02 [scrapy.core.engine] DEBUG: Crawled (400) <GET https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1> (referer: None)
2017-03-02 18:56:02 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <400 https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1>: HTTP status code is not handled or not allowed

Upvotes: 2

Views: 5335

Answers (1)

parik
parik

Reputation: 2415

As you can read in doc, according to the HTTP standard, successful responses are those whose status codes are in the 200-300 range.

If you still want to process response codes outside that range, you can specify which response codes the spider is able to handle using the handle_httpstatus_list spider attribute or HTTPERROR_ALLOWED_CODES setting.

So you should add it in your code 

class XueqiuSpider(scrapy.Spider):
    handle_httpstatus_list = [400]

Upvotes: 1

Related Questions