CODEWITHSUNDEEP
pythonscrapy
Santy
Santy

Reputation: 11

405 error in scrapy python, How can resolve it?

I am getting below Error, when I have run my spider

[scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-30 01:18:36 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-12-30 01:18:37 [scrapy.core.engine] DEBUG: Crawled (405) <GET https://www.propertyguru.com.sg/robots.txt> (referer: None)
2018-12-30 01:18:37 [scrapy.core.engine] DEBUG: Crawled (405) <GET https://www.propertyguru.com.sg/> (referer: None)
2018-12-30 01:18:38 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <405 https://www.propertyguru.com.sg/>: HTTP status

code is not handled or not allowed

Upvotes: 0

Views: 942

Answers (1)

user8527266
user8527266

Reputation:

You'll need to include User-Agent and cookies in your request:

def start_requests(self):
    headers = {'User-Agent': 'your user agent'}
    cookies = {'cookie-key': 'cookie-value'}
    yield scrapy.Request(
        url='https://www.propertyguru.com.sg/',
        method='GET',
        headers=headers,
        cookies=cookies,
        callback=self.parse,
        errback=self.handle_err,
    )

To get User-Agent and cookies, open up google chorme's developer console and type:

navigator.userAgent for User-Agent

document.cookie for cookies

Upvotes: 2

Related Questions