CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingscrapy
Programmer
Programmer

Reputation: 418

failed to create json file in scrapy

I am trying to save scrapped data in json file. I have used scrapy to scrap the data from web.

here is my spider code.

import scrapy
import json

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['quotes.toscrape.com/page/1/']
    start_urls = ['http://quotes.toscrape.com/page/1//']

    def parse(self, response):
        with open('quotes.json', 'a') as f:
            for quote in response.css('div.quote'):
                json.dump({
                     'text' : quote.css('span:text::text').extract_first(),
                     'author': quote.css('small.author::text').extract_first(),
                     'tags': quote.css('div.tags a.tag::text').extract(),
                     }, f)

When I execute this command scrapy crawl quotes, it finished successfully, but json file is not being created.

Please tell me what I'm missing here.

Upvotes: 0

Views: 213

Answers (2)

Krishna
Krishna

Reputation: 198

You need to add an additional parameter to your crawl command like:

scrapy crawl quotes -o output.json

json will be saved in the current folder from where you will execute the above command.

Upvotes: 2

Alejandro Almira
Alejandro Almira

Reputation: 43

If you want to store the json programmatically you should use Item Pipelines

Upvotes: 0

Related Questions