CODEWITHSUNDEEP
python-3.ximporterrorscrapy
mpj
mpj

Reputation: 357

Can't restart scrapy spider

I have scrapy project (Scrapy 1.4, Python 3.x). I want to rerun my spider when condition isn't fulfilled. I do it in my spider_closed function

def __init__(self):
    dispatcher.connect(self.spider_closed, signals.spider_closed)

def spider_closed(self):
    if sth1 != sth2:
        RESTART SPIDER
    else:
        FINISH SPIDER

I tried every possible method: run spider using os command, run spider in bash script, run spider using CrawlerProcess, etc ... but every time I got error

ImportError: No module named ~my project name~

Could anybody help?

Upvotes: 1

Views: 306

Answers (1)

Umair Ayub
Umair Ayub

Reputation: 21201

ImportError: No module named ~my project name~

means it didnt find the path/folder of your project, you have to CD into that project folder before running spider using bash or whatever.

HERE IS WHAT I DID WHEN I WAS IN SAME SCENARIO

I installed ScrapyD and then inside close method, I started my scraper again using scrapyD's schedule endpoint.

def spider_closed(self, spider):

    resp = requests.post("http://localhost:6800/schedule.json", data={'project': "default", 'spider': self.name})
    logging.info(resp.text)

Upvotes: 1

Related Questions