CODEWITHSUNDEEP
pythonscrapy
yayu
yayu

Reputation: 8088

Scrapy: how to run two crawlers one after another?

I have two spiders within the same project. One of them depends on the other running first. They use different pipelines. How can I make sure they are run sequentially?

Upvotes: 3

Views: 4216

Answers (1)

foolcage
foolcage

Reputation: 1104

Just from the doc:https://doc.scrapy.org/en/1.2/topics/request-response.html

Same example but running the spiders sequentially by chaining the deferreds:

from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging

class MySpider1(scrapy.Spider):
    # Your first spider definition
    ...

class MySpider2(scrapy.Spider):
    # Your second spider definition
    ...

configure_logging()
runner = CrawlerRunner()

@defer.inlineCallbacks
def crawl():
    yield runner.crawl(MySpider1)
    yield runner.crawl(MySpider2)
    reactor.stop()

crawl()
reactor.run() # the script will block here until the last crawl call is finished

Upvotes: 4

Related Questions