CODEWITHSUNDEEP
pythonscrapyweb-crawler
kliron
kliron

Reputation: 4663

Importing scrapy.conf.settings generates error

Using scrapy version 0.16. Trying to create a standalone spider runnable from a script as per this gist. Importing the above generates this:

    Traceback (most recent call last):
  File "./spiderctl.py", line 8, in <module>
    from scrapy.conf import settings
  File "/usr/local/lib/python2.6/dist-packages/scrapy/conf.py", line 4, in <module>
    from scrapy.project import crawler
ImportError: cannot import name crawler

The file conf.py contains the line:

from scrapy.project import crawler

But the file scrapy.project is only a comment that the module is deprecated and users should instead implement the from_crawler class method. How do I implement this in the context of the code above?

Upvotes: 1

Views: 2638

Answers (2)

Pablo Hoffman
Pablo Hoffman

Reputation: 1540

You can find the answer to this question in this new FAQ:

Basically, you need to access the crawler differently, using the from_crawler class method instead of importing crawler from scrapy.project.

Upvotes: 1

warvariuc
warvariuc

Reputation: 59604

I think you an older version of scrapy lying around, as the most recent version doesn't have the line you mention:

https://github.com/scrapy/scrapy/blob/master/scrapy/conf.py

UPDATE:

Well, it was there: https://github.com/scrapy/scrapy/commit/e8e5a62c20b3d217cc55038f30e495ca183bbba7#L1R0

Try to install the most recent version and see if it works:

python setup.py install

Upvotes: 1

Related Questions