CODEWITHSUNDEEP
pythonweb-crawlerscrapy
Jason Youk
Jason Youk

Reputation: 802

Scrapy: no route to host and persistent support enabled

If I'm running a crawler with persistent support enabled and I temporarily loose internet connection. Will the crawler retry the URLs that get a no route to host error during the temporary internet loss?

Upvotes: 1

Views: 1041

Answers (1)

R. Max
R. Max

Reputation: 6710

Yes.

Scrapy uses an HTTP 1.1 client which have persistent support by default, and under the hood (thanks to Twisted) this uses a pool of persistent connections with automatic retry when the connection is lost.

Besides that, when Scrapy gets a connection error for a request (timeout, dns error, no route, etc), the RetryMiddleware takes care of retrying the request. See http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.contrib.downloadermiddleware.retry

Upvotes: 2

Related Questions