CODEWITHSUNDEEP
pythonscrapy
Noah R
Noah R

Reputation: 5477

Error Using Python's "Scrapy"

I installed the newest version of Scrapy on Python 2.7 (Yes, I still use this version) and am getting an error while running Scrapy's tutorial script. The code I'm running is:

from scrapy.spider import BaseSpider

class DmozSpider(BaseSpider):
    name = "dmoz.org"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        filename = response.url.split("/")[-2]
        open(filename, 'wb').write(response.body)

The error is:

Traceback (most recent call last):
  File "C:/Users/Rainey/Desktop/dmoz/spiders/dmoz_spider.py", line 1, in <module>
    from scrapy.spider import BaseSpider
  File "C:\Python27\lib\site-packages\scrapy\spider.py", line 7, in <module>
    from scrapy import log
  File "C:\Python27\lib\site-packages\scrapy\log.py", line 10, in <module>
    from twisted.python import log
ImportError: No module named twisted.python

The tutorial script is on my desktop. Will this effect and should I have it in a different place?

Upvotes: 1

Views: 2709

Answers (2)

Monster Z
Monster Z

Reputation: 147

You did not install scrapy successfully on windows. following steps should work for you to install scrapy 0.14(latest) on windows.

  1. Install lxml-2.3 using windows installer here. I tried libxml but not work.
  2. Install Twisted-12.0.0 using windows installer here
  3. Install zope.interface with Mingw32 or not.
  4. Install pyOpenSSL 0.13 using windows installer here
  5. run "pip install scrapy", you are ready to go!

Hope it helps.

Upvotes: 2

Gustavo Breeze
Gustavo Breeze

Reputation: 66

The error message 'ImportError: No module named twisted.python' indicates that you are missing the twisted module.

Install twisted using a package manager like pip or download a Windows binary.

Upvotes: 5

Related Questions