CODEWITHSUNDEEP
pythonscrapyscrapyd
Steve
Steve

Reputation: 9

Scrapy Connection Refused

Hi I have been using Scrapy & Scrapyd for some time now and recently I have encountered a really weird issue. All my spiders will work for an unknown amount of time (days), then it just stopped working. When I checked the logs it says Connection refused 111 on all spiders, I just have to restart my Ubuntu instance then all spiders will work again (before failing again after x number of days).

Any help guys?

Upvotes: 0

Views: 369

Answers (1)

Ike
Ike

Reputation: 1059

Are you using scrapy-splash? I have noticed that splash will crash after a while (due to some memory issues).

If you are using splash with docker, you can run 

docker ps -a

to see when splash failed and why (Error 137 is a memory issue, apparently).

Upvotes: 1

Related Questions