CODEWITHSUNDEEP
pythonscrapy
Simeoni Peraparota
Simeoni Peraparota

Reputation: 299

spider_closed function is not fired

I want to log some data when the spider finishes working.

I make this function inside my spider.

def spider_closed(self, spider):
        log.msg('The spider {0} finished working on'.format(self.name), level=log.INFO)

But this function is not getting fired.

I am using scrapy 0.20 with python 2.7

Please note that I can log to my log anything and actually I am logging a lot of data that I can see in my log file. so the logging is working with me.

Edit

I guess that I have to dispatch a signal signals.spider_closed but I don't know how to do that.

Upvotes: 4

Views: 696

Answers (1)

alecxe
alecxe

Reputation: 473903

You should attach your method as a receiver to the SignalManager signal dispatcher:

from scrapy.xlib.pydispatch import dispatcher
from scrapy import signals

class MySpider(Spider):
    def __init__(self, *args, **kwargs):
        dispatcher.connect(self.spider_closed, signal=signals.spider_closed)
        super(MySpider, self).__init__(*args, **kwargs)

    def spider_closed(self, spider):
        log.msg('The spider {0} finished working on'.format(self.name), level=log.INFO)

Also see relevant threads:

Upvotes: 5

Related Questions