unable to create scrapy project in python

Question

I have tried to create scrapy project after installing scrapy module. Ended up with below error. I have searched for some solution in stackoverflow. But could not find anything.

In directory test1 directory is got created with some files and folders which look like below. In that I can't find settings.py.

test1
|-test1
    |-__pycache__
    |-spiders
        |-__pycache__
        |-__init__.py
    |-__init__.py
|-scrapy.cfg

Below is the error message returned while creating scarpy project.

    C:\Users\SIMBU\Personal\MyWork\PyPgm>scrapy startproject test1
Traceback (most recent call last):
  File "d:\myFld\python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "d:\myFld\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\myFld\Python36\Scripts\scrapy.exe\__main__.py", line 9, in <module>
  File "d:\myFld\python36\lib\site-packages\scrapy\cmdline.py", line 149, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "d:\myFld\python36\lib\site-packages\scrapy\cmdline.py", line 89, in _run_print_help
    func(*a, **kw)
  File "d:\myFld\python36\lib\site-packages\scrapy\cmdline.py", line 156, in _run_command
    cmd.run(args, opts)
  File "d:\myFld\python36\lib\site-packages\scrapy\commands\startproject.py", line 109, in run
    ProjectName=string_camelcase(project_name))
  File "d:\myFld\python36\lib\site-packages\scrapy\utils\template.py", line 9, in render_templatefile
    with open(path, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'test1\\test1\\settings.py.tmpl'

EDIT : After scrapy installation, project creation is never successful. settings file in (D:\myFld\Python36\Lib\site-packages\scrapy\settings\default_settings.py) has directory value as follows

TEMPLATES_DIR = abspath(join(dirname(__file__), '..', 'templates'))

My pip shows results as below.

C:\Users\SIMBU>pip show scrapy
Name: Scrapy
Version: 1.4.0
Summary: A high-level Web Crawling and Web Scraping framework
Home-page: http://scrapy.org
Author: Pablo Hoffman
Author-email: pablo@pablohoffman.com
License: BSD
Location: d:\myFld\python36\lib\site-packages
Requires: queuelib, lxml, w3lib, cssselect, Twisted, parsel, PyDispatcher, service-identity, pyOpenSSL, six
Required-by:

Answer 1

Reinstalling scrapy worked for me. Here is the steps I have follwed to resolve this issue.

C:\Users\SIMBU>pip uninstall scrapy
Uninstalling Scrapy-1.4.0:
  Would remove:
    d:\myFld\python36\lib\site-packages\scrapy-1.4.0.dist-info\*
    d:\myFld\python36\lib\site-packages\scrapy\*
    d:\myFld\python36\scripts\scrapy.exe
Proceed (y/n)? y
  Successfully uninstalled Scrapy-1.4.0

C:\Users\SIMBU>
C:\Users\SIMBU>pip install scrapy
Collecting scrapy
  Downloading https://files.pythonhosted.org/packages/5d/12/a6197eaf97385e96fd8ec56627749a6229a9b3178ad73866a0b1fb377379/Scrapy-1.5.1-py2.py3-none-any.whl (249kB)
    100% |████████████████████████████████| 256kB 757kB/s
Requirement already satisfied: w3lib>=1.17.0 in d:\myFld\python36\lib\site-packages (from scrapy) (1.18.0)
Requirement already satisfied: service-identity in d:\myFld\python36\lib\site-packages (from scrapy) (17.0.0)
Requirement already satisfied: lxml in d:\myFld\python36\lib\site-packages (from scrapy) (4.1.1)
Requirement already satisfied: PyDispatcher>=2.0.5 in d:\myFld\python36\lib\site-packages (from scrapy) (2.0.5)
Requirement already satisfied: cssselect>=0.9 in d:\myFld\python36\lib\site-packages (from scrapy) (1.0.1)
Requirement already satisfied: Twisted>=13.1.0 in d:\myFld\python36\lib\site-packages (from scrapy) (17.9.0)
Requirement already satisfied: parsel>=1.1 in d:\myFld\python36\lib\site-packages (from scrapy) (1.2.0)
Requirement already satisfied: queuelib in d:\myFld\python36\lib\site-packages (from scrapy) (1.4.2)
Requirement already satisfied: pyOpenSSL in d:\myFld\python36\lib\site-packages (from scrapy) (18.0.0)
Requirement already satisfied: six>=1.5.2 in d:\myFld\python36\lib\site-packages (from scrapy) (1.11.0)
Requirement already satisfied: pyasn1-modules in d:\myFld\python36\lib\site-packages (from service-identity->scrapy) (0.2.1)
Requirement already satisfied: attrs in d:\myFld\python36\lib\site-packages (from service-identity->scrapy) (17.3.0)
Requirement already satisfied: pyasn1 in d:\myFld\python36\lib\site-packages (from service-identity->scrapy) (0.4.2)
Requirement already satisfied: constantly>=15.1 in d:\myFld\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (15.1.0)
Requirement already satisfied: zope.interface>=4.0.2 in d:\myFld\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (4.4.3)
Requirement already satisfied: hyperlink>=17.1.1 in d:\myFld\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (17.3.1)
Requirement already satisfied: incremental>=16.10.1 in d:\myFld\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (17.5.0)
Requirement already satisfied: Automat>=0.3.0 in d:\myFld\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (0.6.0)
Requirement already satisfied: cryptography>=2.2.1 in d:\myFld\python36\lib\site-packages (from pyOpenSSL->scrapy) (2.3.1)
Requirement already satisfied: setuptools in d:\myFld\python36\lib\site-packages (from zope.interface>=4.0.2->Twisted>=13.1.0->scrapy) (40.4.3)
Requirement already satisfied: cffi!=1.11.3,>=1.7 in d:\myFld\python36\lib\site-packages (from cryptography>=2.2.1->pyOpenSSL->scrapy) (1.11.2)
Requirement already satisfied: idna>=2.1 in d:\myFld\python36\lib\site-packages (from cryptography>=2.2.1->pyOpenSSL->scrapy) (2.6)
Requirement already satisfied: asn1crypto>=0.21.0 in d:\myFld\python36\lib\site-packages (from cryptography>=2.2.1->pyOpenSSL->scrapy) (0.23.0)
Requirement already satisfied: pycparser in d:\myFld\python36\lib\site-packages (from cffi!=1.11.3,>=1.7->cryptography>=2.2.1->pyOpenSSL->scrapy) (2.18)
Installing collected packages: scrapy
Successfully installed scrapy-1.5.1

C:\Users\SIMBU>
C:\Users\SIMBU>scrapy startprojct test
Scrapy 1.5.1 - no active project

Unknown command: startprojct

Use "scrapy" to see available commands

C:\Users\SIMBU>scrapy startproject test
Error: Module 'test' already exists

C:\Users\SIMBU>scrapy startproject test2
New Scrapy project 'test2', using template directory 'd:\\myFld\\python36\\lib\\site-packages\\scrapy\\templates\\project', created in:
    C:\Users\SIMBU\test2

You can start your first spider with:
    cd test2
    scrapy genspider example example.com

C:\Users\SIMBU>

Answer 2

It's trying to find template files inside the project it's creating, when it should be looking at the scrapy install location.

Did you change the TEMPLATES_DIR setting at any point before this? Have you created projects previously using your current installation?

(Apologies to the community: I know clarity should be asked for in a comment but I don't have enough rep and I'm trying to help)