CODEWITHSUNDEEP
pythonscrapy
user891260
user891260

Reputation: 15

scrapy is using HTTP 1.0 by default

It looks like Scrapy is using HTTP 1.0 by default. Is there a setting to make it use HTTP 1.1 to send request?

Thanks.

Upvotes: 0

Views: 741

Answers (1)

pricco
pricco

Reputation: 2783

From http://dev.scrapy.org/wiki/ScrapyRecipes:

How to spoof requests to be HTTP 1.1 compliant You can do this by overriding the Scrapy HTTP Client Factory, with the following (undocumented) setting:

DOWNLOADER_HTTPCLIENTFACTORY = 'myproject.downloader.HTTPClientFactory'

Here's a possible implementation of myproject.downloader module:

from scrapy.core.downloader.webclient import ScrapyHTTPClientFactory, ScrapyHTTPPageGetter

class PageGetter(ScrapyHTTPPageGetter):

    def sendCommand(self, command, path):
        self.transport.write('%s %s HTTP/1.1\r\n' % (command, path))

class HTTPClientFactory(ScrapyHTTPClientFactory):

    protocol = PageGetter

Upvotes: 4

Related Questions