CODEWITHSUNDEEP
pythonweb-crawlerscrapy
Computer's Guy
Computer's Guy

Reputation: 5363

Scrapy: ImportError: No module named project_name.settings

i'm trying to make a script that runs many spiders but i'm getting ImportError: No module named project_name.settings

my script looks like this:

import os
os.system("scrapy crawl spider1")
os.system("scrapy crawl spider2")
....
os.system("scrapy crawl spiderN")

My settings.py

# -*- coding: utf-8 -*-

# Scrapy settings for project_name
#
# For simplicity, this file contains only the most important settings by
# default. All the other settings are documented here:
#
#     http://doc.scrapy.org/en/latest/topics/settings.html
#

BOT_NAME = 'project_name'

ITEM_PIPELINES = {
    'project_name.pipelines.project_namePipelineToJSON': 300,
    'project_name.pipelines.project_namePipelineToDB': 800
}

SPIDER_MODULES = ['project_name.spiders']
NEWSPIDER_MODULE = 'project_name.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'project_name (+http://www.yourdomain.com)'

And my spiders look like any normal spider, quite simple ones actually...

import scrapy
from scrapy.crawler import CrawlerProcess
from Projectname.items import ProjectnameItem

class ProjectnameSpiderClass(scrapy.Spider):
    name = "Projectname"
    allowed_domains = ["Projectname.com"]

    start_urls = ["...urls..."]


    def parse(self, response):
        item = ProjectnameItem()

I gave them generic names but you get the idea, is there a way to solve this error?

Upvotes: 2

Views: 2750

Answers (1)

Computer's Guy
Computer's Guy

Reputation: 5363

Edit 2018:

You need to run the spider from the project folder, meaning that the os.system("scrapy crawl spider1") has to be run from the folder with the spider1.

Or you can do as I did in the past, putting all the code in a single file (old answer, not recommended by me anymore, but still useful and decent solution)

Well, in case someone comes up to this question I finally used a heavily modified version of this https://gist.github.com/alecxe/fc1527d6d9492b59c610 provided by alexce in another question. Hope this helps.

Upvotes: 1

Related Questions