CODEWITHSUNDEEP
scrapy
codehia
codehia

Reputation: 184

Scrapy spider not consistently terminating with use of CloseSpider extension

i tried using CLOSESPIDER_TIMEOUT extension in the settings to kill the spiders which are running beyond 3 hours. 

CLOSESPIDER_TIMEOUT = 3 * 60 * 60

Although the spiders receive the close timeout request, It never actually stops the spider, which keeps on running.

Any ideas on what's wrong in this case?

Upvotes: 1

Views: 834

Answers (1)

Wim Hermans
Wim Hermans

Reputation: 2116

If your spider get the close timeout request, the extension seems to be working. It doesn't look like anything is wrong, but you might have to wait a bit before the spider fully closes, as he will first finish the already scheduled requests before shutting down completely.

Upvotes: 2

Related Questions