CODEWITHSUNDEEP
pythonweb-crawlerscrapy
akhter wahab
akhter wahab

Reputation: 4085

Write text file to pipeline

I have multiple spiders in a single scrapy project.

I want to write a separate output text file for each spider with spider name and time stamp.

When I had a single spider I was creating file in __init method but now I am trying like this, upromise will generate two output files while other will only one. 

class MallCrawlerPipeline(object):

def spider_opened(self, spider):
    self.aWriter = csv.writer(open('../%s_%s.txt' % (spider.name, datetime.now().strftime("%Y%m%d_%H%M%S")), 'wb'),
        delimiter=',', quoting=csv.QUOTE_MINIMAL)
    self.aWriter.writerow(['mall', 'store', 'bonus', 'per_action', 'more_than','up_to', 'deal_url', 'category'])

    if 'upromise' in spider.name:
        self.cWriter = csv.writer(
            open('../%s_coupons_%s.txt' % (spider.name, datetime.now().strftime("%Y%m%d_%H%M%S")), 'wb'),
            delimiter=',', quoting=csv.QUOTE_MINIMAL)
        self.cWriter.writerow(['mall', 'store', 'bonus', 'per_action', 'more_than','up_to', 'deal_url', 'category'])

def process_item(self, item, spider):
    self.aWriter.writerow([item['mall'], item['store'], item['bonus'], item['per_action'],
                           item['more_than'], item['up_to'], item['deal_url'], item['category']])

    return item

But I am facing this bug:

 File "C:\Users\akhter\Dropbox\akhter\mall_crawler\mall_crawler\pipelines.py", line 24, in process_item
    self.aWriter.writerow([item['mall'], item['store'], item['bonus'], item['per_action'],
exceptions.AttributeError: 'MallCrawlerPipeline' object has no attribute 'aWriter'

Any help would be appreciated. Thanks in advance.

Upvotes: 1

Views: 1088

Answers (2)

akhter wahab
akhter wahab

Reputation: 4085

thanks guys i found an answer i just need to give a signal otherwise spider_opened never call like this in init method. still open for suggestions 

def __init__(self):
    dispatcher.connect(self.spider_opened, signals.spider_opened)

Upvotes: 0

Eduardo Ivanec
Eduardo Ivanec

Reputation: 11862

Are you sure you're always running obj.spider_opened(...) before obj.process_item(...)? It seems you're not, as after the first method call that attribute should have been added to the object.

If the first method call is always needed perhaps it makes sense to move it to __init__ or call it from there, by the way.

Upvotes: 1

Related Questions