CODEWITHSUNDEEP
pythonjson
user3168180
user3168180

Reputation: 65

Run Scrapy spider via script & configure the settings for output file

I have written a spider in scrapy & running it in a python script (not scrapy cmd prompt). I want to configure settings, so as to get the Scrapped data in a particular file (say output.json).

I can get the result if I run following command on the prompt:"scrapy crawl myspider -o scrapedData.json -t json"

But I want the same output by running a script not via cmdline tool.

Thanks for any help!

Upvotes: 3

Views: 830

Answers (1)

davegallant
davegallant

Reputation: 533

settings = get_project_settings()
settings.overrides['FEED_URI'] = 'dealsOutput.json'
settings.overrides['FEED_FORMAT'] = 'json'

spider = dealsSpider()
crawler = Crawler(settings)

I found by looking at this code: https://github.com/scrapy/scrapy/blob/master/scrapy/commands/crawl.py#L34

Upvotes: 3

Related Questions