CODEWITHSUNDEEP
pythonscrapy
DevOps
DevOps

Reputation: 438

Item not reaching pipeline

I am new to python and scrapy. I am not getting item data in pipeline. Nothing is being written in csv. Error is

DmozSpider' object has no attribute getitem

Any help will be appreciated:

spider file

import scrapy
import sys
import os
from tutorial.items import TutorialItem
from pprint import pprint
class DmozSpider(scrapy.Spider):
    name = "myspider"
    allowed_domains = ["www.xyz.co.id"]
    start_urls = ["http://www.xyz.co.id/search?q=abc"]

    def parse(self, response):
        var = response.xpath("//a[@class='img']/@href").extract()[0]
        item = TutorialItem()
        item['title'] = var
        yield item

pipeline file

import csv

class TutorialPipeline(object):

    def __init__(self):
        self.csvwriter = csv.writer(open('items.csv', 'wb'))

    def process_item(self, domain, item):
        print item['title']
        self.csvwriter.writerow([item['title']])
        return item

items file

import scrapy

class TutorialItem(scrapy.Item):
    title = scrapy.Field()
    link = scrapy.Field()
    desc = scrapy.Field()
    price = scrapy.Field()

Settings file

         ITEM_PIPELINES = {
       'tutorial.pipelines.TutorialPipeline': 300,
   }

Upvotes: 0

Views: 73

Answers (2)

dron22
dron22

Reputation: 1233

The definition of your pipeline method process_item() is incorrect. The bug is in the stated parameters self, domain, item. The official description in the documentation is:

process_item(self, item, spider)

Change the method in your class TutorialPipeline accrodingly to:

def process_item(self, item, spider):
    print item['title']
    self.csvwriter.writerow([item['title']])
    return item

Upvotes: 1

Miriam
Miriam

Reputation: 25

Try item.get('title') instead of item['title']

Upvotes: 0

Related Questions