CODEWITHSUNDEEP
pythonscrapy
user11642562
user11642562

Reputation:

Debug while webcraping with scrapy?

I am trying to webscrape a website with scrapy (with a 10s download delay + AUTOTHROTTLE_ENABLED = True + ROBOTSTXT_OBEY = True). When I run the command:

scrape crawl myspider -o mydata.csv

I get this output for multiple "lines" until 405:

[protego] DEBUG: Rule at line 4 without any user agent to enforce it on.

and also:

[scrapy.spidermiddlewares.httperror] INFO: Ignoring response <405 https://www.funda.nl/koop/utrecht/>: HTTP status code is not handled or not allowed

Why can't scrapy scrape this website?

and this is the dumping stats (if useful): 

2019-11-03 00:33:55 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 417,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 23402,
 'downloader/response_count': 2,
 'downloader/response_status_count/405': 2,
 'elapsed_time_seconds': 13.42723,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 11, 2, 23, 33, 55, 12777),
 'httperror/response_ignored_count': 1,
 'httperror/response_ignored_status_count/405': 1,
 'log_count/DEBUG': 61,
 'log_count/INFO': 11,
 'memusage/max': 51818496,
 'memusage/startup': 51818496,
 'response_received_count': 2,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/405': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2019, 11, 2, 23, 33, 41, 585547)}

Upvotes: 5

Views: 8265

Answers (1)

Umair Ayub
Umair Ayub

Reputation: 21261

You need to emulate exactly same request as a real browser does

headers = {
    'Connection': 'keep-alive',
    'Cache-Control': 'max-age=0',
    'DNT': '1',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36',
    'Sec-Fetch-User': '?1',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-Fetch-Mode': 'navigate',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.9',
}

yield scrapy.Request('https://www.funda.nl/koop/utrecht/', headers=headers)

Upvotes: 7

Related Questions