CODEWITHSUNDEEP
python-2.7anacondascrapy
F Ali
F Ali

Reputation: 23

Running scrapy spider with Anaconda

Hello I am trying to complete the tutorial found at:

https://doc.scrapy.org/en/latest/intro/tutorial.html

I have completed all the steps up to "Our first Spider" without error however when I try to run the spider I get the following error message when I run "scrapy crawl quotes":

File "//anaconda/lib/python2.7/site-packages/scrapy/spiderloader.py", line 43, in load raise KeyError("Spider not found: {}".format(spider_name)) KeyError: 'Spider not found: quotes'

So for some reason, the spider manager is not finding the spider. However, When I navigate directly to the spider directory I can find the spider without error.

Upvotes: 2

Views: 2211

Answers (1)

Rejected
Rejected

Reputation: 4491

The Scrapy command line option "crawl" requires a Scrapy project in order to find the specified spider, and start the crawl (see documentation here, in which "Requires Project: yes" is listed).

Realistically, Scrapy would have no way of knowing the location of all Scrapy spiders on your system, picking one, and running it. For options that require a Project, you must be in the directory of the Scrapy project, which will have the file scrapy.cfg in it.

If you don't wish to use "crawl", you can use "runspider" instead, and specify the Python file containing the spider.

Upvotes: 1

Related Questions