CODEWITHSUNDEEP
pythonpython-2.7web-scrapingscrapy
Umair Ayub
Umair Ayub

Reputation: 21261

Call a Spider's method from Pipeline - Python Scrapy

This is Spider

class TicketsSpider(scrapy.Spider):

    def __set_last_start_date(self, dateString):
        #code here

This is Pipeline

class TicketsPipeline(object):

    def spider_closed(self, spider):
        spider.__set_last_start_date(spider.lastAdScrapedDate)

    @classmethod
    def from_crawler(cls, crawler):
        pipeline = cls()
        crawler.signals.connect(pipeline.spider_closed, signals.spider_closed)
        return pipeline

I want to call __set_last_start_date() function from spider_closed() method. But I am getting this error

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 149, in maybeDeferred
    result = f(*args, **kw)
  File "build/bdist.linux-x86_64/egg/pydispatch/robustapply.py", line 55, in robustApply
  File "tickets/pipelines.py", line 236, in spider_closed
    spider.__set_last_start_date(spider.lastAdScrapedDate)
AttributeError: 'TicketsSpider' object has no attribute '_TicketsPipeline__set_last_start_date'
2016-12-13 02:49:53 [scrapy] INFO: Dumping Scrapy stats:

I can assure you that I can get spider.lastAdScrapedDate but I am unable to call spider.__set_last_start_date

Upvotes: 1

Views: 775

Answers (1)

pennyliangliping
pennyliangliping

Reputation: 93

Names, in a class, with double underscore leading are intended to be private.

Any identifier of the form __spam (at least two leading underscores, at most one trailing underscore) is textually replaced with _classname__spam, where classname is the current class name with leading underscore(s) stripped.

Please see this for detail.

option 1. You can rename your method name, not use double underscore leading.

option 2. If you want to keep your method name, then call it in this way, but I don't think it's a good idea:

    def close_spider(self, spider):
        spider._TicketsSpider__set_last_start_date(spider.lastAdScrapedDate)

Upvotes: 3

Related Questions