CODEWITHSUNDEEP
pythonscrapy
zappy
zappy

Reputation: 33

No module named 'scrapy.conf'

I'm trying to execute Twitter scraper code from: https://github.com/jonbakerfish/TweetScraper

When I run the command scrapy list in the command prompt to ensure the scraper is properly set up, I get the following error:

I have tried to install scrapy. Still no luck.

from scrapy.conf import settings ModuleNotFoundError: No module named 'scrapy.conf'".

Upvotes: 2

Views: 3024

Answers (2)

joshijai2
joshijai2

Reputation: 167

This is old but the solution provided doesn't work.

The error from scrapy.conf import settings ModuleNotFoundError: No module named 'scrapy.conf' is due to the versioning as it is depreciated for Scrapy > 1.6

You can access the settings in either of the 2 ways:

  1. The solution provided in the documentation says -

The scrapy.conf module is removed, use Crawler.settings

Here's a code sample using the above method. Source: feedexport.py

@implementer(IFeedStorage)
class BlockingFeedStorage:

    def open(self, spider):
        path = spider.crawler.settings['FEED_TEMPDIR']
  1. You can also access the settings using -

from scrapy.utils.project import get_project_settings

Here's a code sample using the above method.

from scrapy.utils.project import get_project_settings

settings = get_project_settings()

Upvotes: 0

Alex Weavers
Alex Weavers

Reputation: 710

That code seems to have a scrapy.cfg file and is 3 years old. Might be that you have a newer version of scrapy and it's expecting a different config setup.

It changed to:

from scrapy import settings

Upvotes: 1

Related Questions