CODEWITHSUNDEEP
scrapy
Sarvesh
Sarvesh

Reputation: 83

How do I stop crawling and close spider based on condition?

I have a spider which fetches the latest url based on a particular date range from a paginated webpage. When it gets all latest urls, my spider has to be closed.
How to close a spider?
I referred question : Force stop the spider
But raising an exception to close the spider is not pleasing to me.
Is there any other way I could achieve the same?

Upvotes: 3

Views: 1586

Answers (1)

dataisbeautiful
dataisbeautiful

Reputation: 546

You should use the Close Spider extension.

The conditions for closing a spider can be configured through the following settings:

CLOSESPIDER_TIMEOUT CLOSESPIDER_ITEMCOUNT CLOSESPIDER_PAGECOUNT CLOSESPIDER_ERRORCOUNT

Upvotes: 2

Related Questions