CODEWITHSUNDEEP
pythonweb-scrapingscrapy
NeDark
NeDark

Reputation: 1294

Scrapy disable retry middleware

I commented the line in settings.py but it continues being enabled.

DOWNLOADER_MIDDLEWARES = {  
       #'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
       'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
    }

At the start of the program it loads a lot of middlewares I didn't enable

2019-08-09 10:43:37 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']

Am I missing something? Is there a way to disable it?

Upvotes: 3

Views: 2155

Answers (1)

Jan
Jan

Reputation: 43169

According to the documentation, the DOWNLOADER_MIDDLEWARES is merged with DOWNLOADER_MIDDLEWARES_BASE. In the latter, the option scrapy.downloadermiddlewares.httpproxy.RetryMiddleware is enabled by default.

So either write

DOWNLOADER_MIDDLEWARES = {  
       'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
       #                                                     ^^^
       'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
    }

Or look into DOWNLOADER_MIDDLEWARES_BASE. See their documentation for more details.

Upvotes: 4

Related Questions