CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingscrapy
cs95
cs95

Reputation: 402814

ScrapyDeprecationWarning: Module `scrapy.conf` is deprecated, use `crawler.settings` attribute instead

I am using scrapy 1.5 and am programmatically trying to run a scrapy crawler process through a python script. As a part of this, I need to import the crawler settings and override some of its parameters.

I found this import statement does what I need:

from scrapy.conf import settings

The problem is, this works, but also produces the following deprecation warning:

ScrapyDeprecationWarning: Module `scrapy.conf` is deprecated, use `crawler.settings` attribute instead

So I'm guessing this is for some older version. Following the warning, I tried to access scrapy.crawler.settings, but this does not exist (or, I couldn't find it after some searching).

How can I resolve this warning?

Upvotes: 3

Views: 2600

Answers (1)

cs95
cs95

Reputation: 402814

This is only mentioned in passing, but I found the right way to do this in the official documentation.

You can use get_project_settings to get a Settings instance with the project settings:

from scrapy.utils.project import get_project_settings
SETTINGS = get_project_settings() 

SETTINGS
# {'AJAXCRAWL_ENABLED': False, 'AUTOTHROTTLE_DEBUG': False, ...'USER_AGENT': 'Scrapy/1.5.0 (+https://scrapy.org)'}

You can then modify this as needed before passing it to the CrawlerProcess.

Upvotes: 14

Related Questions