CODEWITHSUNDEEP
pythonpython-2.7python-3.xweb-crawlerscrapy
krao
krao

Reputation: 41

Scrapy startproject giving weird error

I'm almost positive this is because of python2.7 vs python3.5 issues but I haven't found a way to fix it.

I run 

    $ scrapy startproject test1

and get this error:

    Traceback (most recent call last):
       File "/Library/Frameworks/Python.framework/Versions/3.5/bin/scrapy", line 9, in <module>
          load_entry_point('Scrapy==1.0.1', 'console_scripts', 'scrapy')()
       File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/cmdline.py", line 122, in execute
          cmds = _get_commands_dict(settings, inproject)
       File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/cmdline.py", line 46, in _get_commands_dict
          cmds = _get_commands_from_module('scrapy.commands', inproject)
       File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/cmdline.py", line 29, in _get_commands_from_module
          for cmd in _iter_command_classes(module):
       File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-     packages/scrapy/cmdline.py", line 21, in _iter_command_classes
    for obj in vars(module).itervalues():
    AttributeError: 'dict' object has no attribute 'itervalues'

Upvotes: 3

Views: 2172

Answers (5)

kame
kame

Reputation: 21990

The solution is:

sudo pip install scrapy==1.1.0rc1

Scrapy is now available for Python3.

Upvotes: 3

Rejected
Rejected

Reputation: 4501

As others have said, you need to use Python 2.7 with Scrapy. Scrapy is heavily reliant on Twisted (although, only a small portion of it), which is not compatible with Python 3.

Since you're planning on running multiple versions of Python on the same system, I'd recommend you look into virtualenv (I'd actually recommend this regardless of what you're doing on the system, it's just THAT helpful). Virtualenv is a tools which allows you to setup and install a "virtual" python installation in its own directory with it's own packages.

A couple good guides can be found here, and here.

Upvotes: 0

Jimmy
Jimmy

Reputation: 2280

python3 , use dict.items() instead of dict.iteritems()

iteritems() was removed in python3, so you can't use this method anymore.

A simple fix would be to do a fresh installation of scrapy in python2.7 (Link your python default to 2.7) Then do a fresh insstalaltion of python-pip which will use python2.7 instead of python3+ as: sudo apt-get install python-pip After pip installion use: sudo pip install Scrapy

To test scrapy do: scrapy fetch http://www.example.com/some/page.html

Upvotes: 0

PascalVKooten
PascalVKooten

Reputation: 21451

As of writing (4th of August 2015), scrapy is simply not supported in python 3. It is often mentioned, but there has never been a full translation.

It has to do with the fact that Twisted (an integral part of Scrapy) is not supported in Python 3.

Upvotes: 2

francisco sollima
francisco sollima

Reputation: 8338

It seems that Python 3.5's dict type no longer has the method 'itervalues'. Unless they fix the issue, you might be forced to use a prior version of Python.

EDIT:

If you want, you could change the code, and replace the function itervalues() with items(). I don't think that would be a good idea, but it can be a workaround if you must use 3.5

Upvotes: 0

Related Questions