CODEWITHSUNDEEP
pythonmysqlweb-scrapingscrapy
Clément
Clément

Reputation: 21

SCRAPY: how to store data into Mysql database

I have a problem with scrapy, while attempting to store data into a mysql database: I get the following error: (screenshot here)

My code in pipelines.py is 

class SQLStorePipeline(object):

    def __init__(self):
        self.dbpool = adbapi.ConnectionPool('localhost', db='python',
                user='root', passwd='', cursorclass=MySQLdb.cursors.DictCursor,
                charset='utf8', use_unicode=True)

    def process_item(self, item, spider):
        # run db query in thread pool
        query = self.dbpool.runInteraction(self._conditional_insert, item)
        query.addErrback(self.handle_error)

        return item

    def _conditional_insert(self, tx, item):
        # create record if doesn't exist. 
        # all this block run on it's own thread
        tx.execute("select * from test where name = %s", (item['name'][0], ))
        result = tx.fetchone()
        if result:
            log.msg("Item already stored in db: %s" % item, level=log.DEBUG)
        else:
            tx.execute(\
                "insert into test (name, price) "
                "values (%s, %s)",
                (item['link'][0],
                 datetime.datetime.now())
            )
            log.msg("Item stored in db: %s" % item, level=log.DEBUG)

    def handle_error(self, e):
        log.err(e)

(I took it from here).

And my parsing class is:

def parse(self, response):
    item = DmozItem()
    item['name'] = response.xpath('//meta[@itemprop="name"]/@content').extract()[0]
    item['price'] = response.xpath('//meta[@itemprop="price"]/@content').extract()[0]
    yield item

I know that this question has already been asked, but I tried all the different answer before asking here, and none of them works...

Does someone can please help me? Thank you in advance!

Upvotes: 1

Views: 1695

Answers (2)

Clément
Clément

Reputation: 21

I found the solution. Actually @alecxe was right, and his remarks led me to the solution.

MySQLdb was simply not installed, and the reason is that its installation failed because I have an accent in my name and that Python could not handle the path because of this.

Once again, thank you very much @alecxe !

Upvotes: 1

alecxe
alecxe

Reputation: 473873

Read the error carefully - it says IndentationError on the following line:

yield item

This means you need to check your indentation to be consistent (4 spaces per indent):

def parse(self, response):
    item = DmozItem()
    item['name'] = response.xpath('//meta[@itemprop="name"]/@content').extract()[0]
    item['price'] = response.xpath('//meta[@itemprop="price"]/@content').extract()[0]
    yield item

And don't mix tab and spaces if this is the case.

Upvotes: 0

Related Questions