CODEWITHSUNDEEP
pythonmoduleweb-scrapingscrapy
Slater Victoroff
Slater Victoroff

Reputation: 21914

Unable to import items in scrapy

I have a very basic spider, following the instructions in the getting started guide, but for some reason, trying to import my items into my spider returns an error. Spider and items code is shown below:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector

from myProject.items import item

class MyProject(BaseSpider):
    name = "spider"
    allowed_domains = ["website.com"]
    start_urls = [
        "website.com/start"
    ]

    def parse(self, response):
        print response.body

from scrapy.item import Item, Field

class ProjectItem(Item):
    title = Field()

When I run this code scrapy either can't find my spider, or can't import my items file. What's going on here? This should be a really example to run right?

Upvotes: 4

Views: 12644

Answers (5)

Hassan Elshiekh
Hassan Elshiekh

Reputation: 1

It seems that your project structure has a folder named package containing the itemsfile, so you need to import it as follows:

from myProject.package.items import item

Upvotes: 0

MazzzyStar
MazzzyStar

Reputation: 65

if the structure like this:

package/
    __init__.py
    subpackage1/
        __init__.py
        moduleX.py
        moduleY.py
    subpackage2/
        __init__.py
        moduleZ.py
    moduleA.py

and if you are in moduleX.py, the way to import other modules can be:

from .moduleY.py import *

from ..moduleA.py import *

from ..subpackage2.moduleZ.py import *

refer:PEP Imports: Multi-Line and Absolute/Relative

Upvotes: 2

Manoj Sahu
Manoj Sahu

Reputation: 2942

you are importing a field ,you must import a class from items.py like from myproject.items import class_name.

Upvotes: 5

warvariuc
warvariuc

Reputation: 59664

I also had this several times while working with scrapy. You could add at the beginning of your Python modules this line:

from __future__ import absolute_import

More info here:

Upvotes: 6

Slater Victoroff
Slater Victoroff

Reputation: 21914

So, this was a problem that I came across the other day that I was able to fix through some trial and error, but I wasn't able to find any documentation of it so I thought I'd put this up in case anyone happens to run into the same problem I did.

This isn't so much an issue with scrapy as it is an issue with naming files and how python deals with importing modules. Basically the problem is that if you name your spider file the same thing as the project then your imports are going to break. Python will try to import from the directory closest to your current position which means it's going to try to import from the spider's directory which isn't going to work.

Basically just change the name of your spider file to something else and it'll all be up and running just fine.

Upvotes: 4

Related Questions