CODEWITHSUNDEEP
pythonscrapy
user2129794
user2129794

Reputation: 2418

Terminate Scrapy if a condition is met

I have written a scraper using scrapy in python. It contains 100 start_urls.

I want to terminate the scraping process once a condition is met. ie terminate scraping of a particular div is found. By terminate I mean it should stop scraping all the urls .

Is it possible

Upvotes: 5

Views: 2647

Answers (1)

paul trmbrth
paul trmbrth

Reputation: 20748

What you're looking for is the CloseSpider exception.

Add the following line somewhere at the top of your source file:

from scrapy.exceptions import CloseSpider

And when you detect that your termination condition is met, simply do something like

        raise CloseSpider('termination condition met')

in your callback method (instead of returning or yielding an Item or Request).

Note that requests that are still in progress (HTTP request sent, response not yet received) will still be parsed. No new request will be processed though.

Upvotes: 14

Related Questions