CODEWITHSUNDEEP
pythonscrapy
thomson zeng
thomson zeng

Reputation: 11

The class in Item can`t be recognized after building a python scrapy program

I try to build a scrapy project according to a book. After using 'scrapy startproject tutorial/cd tutorial/scrapy genspider quotes quotes.toscrape.coom' comands and adding the parse function & changing items, the detail code as fellow:

quotes.py:

import scrapy

from tutorial.items import QuoteItem


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ['http://quotes.toscrape.com/']

    def parse(self, response):
        quotes = response.css('.quote')
        for quote in quotes:
            item = QuoteItem()
            item['text'] = quote.css('.text::text').extract_first()
            item['author'] = quote.css('.author::text').extract_first()
            item['tags'] = quote.css('.tags .tag::text').extract()
            yield item

        next = response.css('.pager .next a::attr(href)').extract_first()
        url = response.urljoin(next)
        yield scrapy.Request(url=url, callback=self.parse)

items.py:

import scrapy


class QuoteItem(scrapy.Item):

    text = scrapy.Field()
    author = scrapy.Field()
    tags = scrapy.Field()

The class QuoteItem cann`t be recognized in quotes.py error prompt picture

And after I changed to 'from tutorial.tutorial.items import QuoteItem' and run 'scrapy crawl quotes', there is another error as fellow: error again

And this caused the results can`t be saved. Someone can help, thanks in advance.

Upvotes: 0

Views: 248

Answers (2)

Newt
Newt

Reputation: 887

From Scrapy Tutorial:

To put our spider to work, go to the project’s top level directory and run: scrapy crawl quotes

In new version Scrapy, please go to the top directory:

scrapy runspider xx/spiders/xxx.py(the full path from top level directory)

In your xxx.py, make sure you import the item like this:

from xx.items import xxItem
# (remember to import the right class name, not always QuoteItem)

Pay attention

It's recommended to run scrapy on linux system or WSL. It does not work well on Win. If you meet the problem like the spider is not found, it may be the system issue.

Upvotes: 0

vijay athithya
vijay athithya

Reputation: 1529

It's working fine with that code!! try using scrapy runspider yourspiderfile.py instead of scrapy crawl quotes.There is no error in the code.

Upvotes: 0

Related Questions