CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Yuseferi
Yuseferi

Reputation: 8680

Get All Spiders Class name in Scrapy

in the older version we could get the list of spiders(spider names ) with following code, but in the current version (1.4) I faced with 

[py.warnings] WARNING: run-all-spiders.py:17: ScrapyDeprecationWarning: CrawlerRunner.spiders attribute is renamed to CrawlerRunner.spider_loader.
for spider_name in process.spiders.list():
    # list all the available spiders in my project

Use crawler.spiders.list():

>>> for spider_name in crawler.spiders.list():
...     print(spider_name)

How Can I get spiders list (and equivalent class names) in Scrapy?

Upvotes: 4

Views: 2826

Answers (1)

Tomáš Linhart
Tomáš Linhart

Reputation: 10210

I'm using this in my utility script for running spiders:

from scrapy import spiderloader
from scrapy.utils import project

settings = project.get_project_settings()
spider_loader = spiderloader.SpiderLoader.from_settings(settings)
spiders = spider_loader.list()
classes = [spider_loader.load(name) for name in spiders]

In you case, it should suffice to rename spiders to spider_loader as suggested by the warning message.

Upvotes: 18

Related Questions