python NameError: name 'file' is not defined in python 3.5

Question

I have Windows 10, with Aptana Studio 3. - I installed python in it (v.3.5.0) with terminal - I installed scrapy 1.0

and now, if I try to run a crawler (scrapy) I get this message:

Traceback (most recent call last):
  File "c:\python3.5\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python3.5\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python3.5\Scripts\scrapy.exe\__main__.py", line 5, in <module>
  File "c:\python3.5\lib\site-packages\scrapy\__init__.py", line 48, in <module>
    from scrapy.spiders import Spider
  File "c:\python3.5\lib\site-packages\scrapy\spiders\__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "c:\python3.5\lib\site-packages\scrapy\http\__init__.py", line 15, in <module>
    from scrapy.http.response.html import HtmlResponse
  File "c:\python3.5\lib\site-packages\scrapy\http\response\html.py", line 8, in <module>
    from scrapy.http.response.text import TextResponse
  File "c:\python3.5\lib\site-packages\scrapy\http\response\text.py", line 13, in <module>
    from scrapy.utils.response import get_base_url
  File "c:\python3.5\lib\site-packages\scrapy\utils\response.py", line 12, in <module>
    from twisted.web import http
  File "c:\python3.5\lib\site-packages\twisted\web\http.py", line 92, in <module>
    from twisted.internet import interfaces, reactor, protocol, address
  File "c:\python3.5\lib\site-packages\twisted\internet\reactor.py", line 38, in <module>
    from twisted.internet import default
  File "c:\python3.5\lib\site-packages\twisted\internet\default.py", line 56, in <module>
    install = _getInstallFunction(platform)
  File "c:\python3.5\lib\site-packages\twisted\internet\default.py", line 50, in _getInstallFunction
    from twisted.internet.selectreactor import install
  File "c:\python3.5\lib\site-packages\twisted\internet\selectreactor.py", line 18, in <module>
    from twisted.internet import posixbase
  File "c:\python3.5\lib\site-packages\twisted\internet\posixbase.py", line 18, in <module>
    from twisted.internet import error, udp, tcp
  File "c:\python3.5\lib\site-packages\twisted\internet\udp.py", line 53, in <module>
    from twisted.internet import base, defer, address
  File "c:\python3.5\lib\site-packages\twisted\internet\base.py", line 23, in <module>
    from twisted.internet import fdesc, main, error, abstract, defer, threads
  File "c:\python3.5\lib\site-packages\twisted\internet\defer.py", line 29, in <module>
    from twisted.python import lockfile, failure
  File "c:\python3.5\lib\site-packages\twisted\python\lockfile.py", line 52, in <module>
    _open = file
NameError: name 'file' is not defined

So far I get that file() is a python 2.x function and it does not work with python 3.x from this Q: python NameError: name 'file' is not defined But the problem is that I do not use file() anywhere in my script. And the message appears to reffer to scrapy default files. So what am I supposed to do? Edit the source files of scrapy? That sounds awfull and way above my knowledge level. So what can I do? I should install python 2.7 but for some reason (I don't remember) I could not install it properly when I tried

So anyone has a solution for this?

Answer 1

You likely need to update Twisted (to v15.4.0 or something similar), since your version is not Python 3 compatible.

As Matthias pointed out, scrapy is being ported to Python 3. However, the error you have is in lockfile.py from Twisted, not scrapy:

File "c:\python3.5\lib\site-packages\twisted\python\lockfile.py", line 52, in <module>
    _open = file

Twisted is a dependency of scrapy. If you look at the history of this file, you'll see that there was a merge that presumably brought in Python 3 compatibility on 9/22/2015. I can tell that you have an older version because only the previous versions have _open = file in that lockfile.py.

All that being said, you probably still need to use Python 2 until the rest of scrapy is updated to be Py3 compatible.

Answer 2

It looks like scrapy is not yet fully ported to Python 3:

https://github.com/scrapy/scrapy/wiki/Python-3-Porting

So yes, use Python 2.