CODEWITHSUNDEEP
pythonpython-3.xscrapy
Trav L
Trav L

Reputation: 15192

Prevent Scrapy from generate empty file when there are no results

Is there a way to prevent scrapy from genrate an empty file when no results yielded from a scrapy spider?

Example usage

scrapy crawl lorem -o /path/to/lorem.json

Versions used:

Upvotes: 2

Views: 271

Answers (2)

Granitosaurus
Granitosaurus

Reputation: 21406

It's a bit of a hack but you can simply delete the file if it's empty.
This could be achieved in a single command in bash: 

scrapy crawl myspider -o test.json && if ! cat test.json; then rm test.json; fi

You could even set up an alias for it.

Upvotes: 1

mizhgun
mizhgun

Reputation: 1888

The only way to do that is to write custom feed storage class based on FileFeedStorage to implement lazy opening of the file when first item is scraped - default FileFeedStorage creates file on start. Then activate it in settings.py:

FEED_STORAGES = {
    '': 'path.to.CustomFileFeedStorage',
    'file': 'path.to.CustomFileFeedStorage',
}

Upvotes: 0

Related Questions