CODEWITHSUNDEEP
pythonscrapy
Murat Demir
Murat Demir

Reputation: 716

Scrapy executing different spiders in different times with reactor

I want to click a button on django website and my spider supposed to start scraping. I try CrawlerProcess but because of I am using different spiders. I decided to use CrawlerRunner. After first Crawling everything is fine but second process giving me this error.

twisted.internet.error.ReactorNotRestartable

I try this topic's solution: ReactorNotRestartable error in while loop with scrapy

So I try solution as following code but the crawler never stop and it says ReactorAlreadyRunning

def test():
    configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
    runner = CrawlerRunner(get_project_settings())
    d = runner.crawl(myspider.My_Spider)
    d.addBoth(lambda _: reactor.stop(stop_after_crawl=False))
    reactor.run()

It is not letting me crawl another spider. So how can I run crawl multiple spiders on different times on my website?

Upvotes: 1

Views: 462

Answers (1)

Murat Demir
Murat Demir

Reputation: 716

This topic helps me to solve the problem. I just need to install crochet and add setup() on top of my code.

Solution link

Upvotes: 1

Related Questions