CODEWITHSUNDEEP
pythonpython-2.7scrapy
PaulGibson
PaulGibson

Reputation: 103

How to create the scrapy project by python3

I use Mac OS, it has python 2.7 and python 3.4. I use the pip install command to to install scrapy in python 2.7. Buy I also use the pip3 install command to install scrapy in python3.4 too...

I read the official documents on scrapy.org, I know that the scrapy just support the python 2.7. When I use the command scrapy startproject tutorial, it will return the errow below.

How can I use the command scrapy startproject tutorial with python 2.7?

  File "/Library/Frameworks/Python.framework/Versions/3.4/bin/scrapy", line 9, in <module>
    load_entry_point('Scrapy==1.1.0dev1', 'console_scripts', 'scrapy')()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Scrapy-1.1.0dev1-py3.4.egg/scrapy/cmdline.py", line 122, in execute
    cmds = _get_commands_dict(settings, inproject)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Scrapy-1.1.0dev1-py3.4.egg/scrapy/cmdline.py", line 46, in _get_commands_dict
    cmds = _get_commands_from_module('scrapy.commands', inproject)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Scrapy-1.1.0dev1-py3.4.egg/scrapy/cmdline.py", line 29, in _get_commands_from_module
    for cmd in _iter_command_classes(module):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Scrapy-1.1.0dev1-py3.4.egg/scrapy/cmdline.py", line 21, in _iter_command_classes
    for obj in vars(module).itervalues():
AttributeError: 'dict' object has no attribute 'itervalues'

Upvotes: 6

Views: 4600

Answers (3)

york zhang
york zhang

Reputation: 1

Actually you can use latest Scrapy to work with python 3. I composed an article about installing Scrapy 3.1.1rc3 for Python3 on Windows and use it in Pycharm. I use Conda for package management and virtual enviroment. Conda works better than pip and easy_install for this job. It should work in Mac with minor changes.

Upvotes: 0

lccurious
lccurious

Reputation: 61

The solution there is, the developer had already provide the scrapy for Python3.x, so you can try the

$ pip install scrapy==1.1.0rc1

And the article is here

Upvotes: 6

H&#229;ken Lid
H&#229;ken Lid

Reputation: 23064

Installing scrapy with pip will put an executable file somewhere in your PATH. Since you installed it two times, the python2 version was probably overwritten. To find this file use the command which scrapy. To see the content of the file use cat $(which scrapy). It probably contains a line the reads something like this: #!/usr/bin/python3.4 causing it to use an incompatible version of python.

To fix this, uninstall the python3 version of scrapy. 

pip3 uninstall scrapy

Then clean the command cache in bash by using hash -r or starting a new terminal session.

If the scrapy command still doesn't work you might have to reinstall the python 2 version of it as well.

pip install scrapy --force-reinstall

Upvotes: 7

Related Questions