Reputation: 184
i tried using CLOSESPIDER_TIMEOUT
extension in the settings to kill the spiders which are running beyond 3 hours.
CLOSESPIDER_TIMEOUT = 3 * 60 * 60
Although the spiders receive the close timeout request, It never actually stops the spider, which keeps on running.
Any ideas on what's wrong in this case?
Upvotes: 1
Views: 834
Reputation: 2116
If your spider get the close timeout request, the extension seems to be working. It doesn't look like anything is wrong, but you might have to wait a bit before the spider fully closes, as he will first finish the already scheduled requests before shutting down completely.
Upvotes: 2