install scrapy on win 7 (64-bit)

Question

I'm trying to install scrapy for python2.6, but it seems not going well. Here is the packages installed:

G:\Python26\Scripts>pip freeze
Scrapy==0.16.4
Twisted==12.3.0
libxml2-python==2.7.7
lxml==2.3.6
pyopenssl==0.13
w3lib==1.2
zope.interface==3.8.0

I also got iconv and zlib. And this is the log when installing scrapy with pip. I don't know what I should do next, am I missing sth? Need instructions, thank you.

win 7 64-bit, Visual C++ installed

C:\Users\d>pip install scrapy
Downloading/unpacking scrapy
  Downloading Scrapy-0.16.4.tar.gz (679kB): 679kB downloaded
  Running setup.py egg_info for package scrapy
    warning: no files found matching 'license.txt' under directory 'scrapy'
    no previously-included directories found matching 'docs\build'
Requirement already satisfied (use --upgrade to upgrade): Twisted>=8.0 in g:\pyt
hon26\lib\site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in g:\pytho
n26\lib\site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in g:\python26\li
b\site-packages\lxml-2.3.6-py2.6-win32.egg (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in g:\python
26\lib\site-packages\pyopenssl-0.13-py2.6-win32.egg (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0
in g:\python26\lib\site-packages\zope.interface-3.8.0-py2.6-win32.egg (from Twis
ted>=8.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in g:\pytho
n26\lib\site-packages (from zope.interface>=3.6.0->Twisted>=8.0->scrapy)
Installing collected packages: scrapy
  Running setup.py install for scrapy
    warning: no files found matching 'license.txt' under directory 'scrapy'
    no previously-included directories found matching 'docs\build'
Successfully installed scrapy
Cleaning up...


C:\Users\d>scrapy
Traceback (most recent call last):
  File "G:\Python26\lib\runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "G:\Python26\lib\runpy.py", line 34, in _run_code
    exec code in run_globals
  File "G:\Python26\lib\site-packages\scrapy\cmdline.py", line 7, in <module>
    from scrapy.crawler import CrawlerProcess
  File "G:\Python26\lib\site-packages\scrapy\crawler.py", line 5, in <module>
    from scrapy.core.engine import ExecutionEngine
  File "G:\Python26\lib\site-packages\scrapy\core\engine.py", line 14, in <modul
e>
    from scrapy.core.downloader import Downloader
  File "G:\Python26\lib\site-packages\scrapy\core\downloader\__init__.py", line
16, in <module>
    from .middleware import DownloaderMiddlewareManager
  File "G:\Python26\lib\site-packages\scrapy\core\downloader\middleware.py", lin
e 7, in <module>
    from scrapy.http import Request, Response
  File "G:\Python26\lib\site-packages\scrapy\http\__init__.py", line 11, in <mod
ule>
    from scrapy.http.request.form import FormRequest
  File "G:\Python26\lib\site-packages\scrapy\http\request\form.py", line 9, in <
module>
    import lxml.html
  File "G:\Python26\lib\site-packages\lxml-2.3.6-py2.6-win32.egg\lxml\html\__ini
t__.py", line 12, in <module>
    from lxml import etree
ImportError: DLL load failed: %1 is not a valid Win32 application

Answer 1

I had the same problem on Windows 10 (64 bit). The error could be reproduced in Python by:

>>> import OpenSSL
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Python27\lib\site-packages\OpenSSL\__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
ImportError: DLL load failed: %1 is not a valid Win32 application.

After installing pyOpenSSL I got the error ImportError: DLL load failed: The operating system cannot run %1., that was solved by copying libeay32.dll and ssleay32.dll to my path (here's my other answer).

Answer 2

I just managed to solve this just now and for people who might stumble upon this, the problem is you need PyWin32. For windows users:

1. pip install PyWin32 here, choose the right python version and 32/64bit. Now in cmd:

   pip install c:\users\desktop\pywin32-220.1-cp27-cp27m-win_amd64.whl

2. Now, install pywin32 system files, run "python.exe Scripts\pywin32_postinstall.py -install" from an elevated command prompt. Run cmd as admin and in the command prompt

   python "C:\Python27\Scripts\pywin32_postinstall.py" -install

Answer 3

I just met with the same problem but I have already solved it.

Please kindly check your installed python package whether the version(32/64bit) is matched your Python installing version.

Since I am now working in a Scrapy project and I installed a 64bit lxml package to my computer, however, I just found out that the default installing version of Python in my computer is 32bit. Therefore, the version is not matched and the problem you posted would be occurred.

Answer 4

My solution is pip install --upgrade PyopenSSL

Answer 5

In my case I did upgrade python which caused this problem.
a simple scrapy upgrade did the trick

pip install --upgrade Scrapy

Also install correct version of pywin : http://doc.scrapy.org/en/latest/intro/install.html#windows

Answer 6

Installing scrapy is not hard. I just completed it on my Windows 7 machine, which is 64-bit:

Install Python 2.7 from http://python.org/download/ : You need to adjust PATH environment variable to include paths to the Python executable and additional scripts. The following paths need to be added to PATH:

C:\Python2.7\;C:\Python2.7\Scripts\;

To update the PATH open a Command prompt and run:

c:\python27\python.exe c:\python27\tools\scripts\win_add2path.py

Close the command prompt window and reopen it so changes take effect, run the following command and check it shows the expected Python version:

python --version

Install pip from https://pip.pypa.io/en/latest/installing.html Now open a Command prompt to check pip is installed correctly:

pip --version

At this point Python 2.7 and pip package manager must be working, let’s install Scrapy:

pip install Scrapy

Check that scrapy works:

scrapy fetch http://www.google.com

All of this was derived from: http://doc.scrapy.org/en/latest/intro/install.html

Answer 7

You need to follow the instructions here (also listed above).

The specific DLL error listed in the question is due (at least in my case) to installing the 32 bit OpenSSL with 64 bit everything else. After I installed the correct OpenSSL (and, incidentally, had to uninstall all of the packages in the link above and start over- order matters for the dependency msi installers), scrapy is working fine on python27, windows7, 64 bit.

Answer 8

Have a read through this guide to installing scrapy on Windows 7. egenix-pyopenssl is listed as a required package, plus installing pywin32 may also help.