Results are not visible when using scrapy with python

Question

I was learning how to scrape with scrapy using python . The result shows that [finished in 1.1s] when i run it through sublimetext, im not able to see what i scrapped. im also not able to see a json file which shows wha i scrapped. can someone help? what i wrote for the code is present below.

import scrapy




class Scrapping(scrapy.Spider):
    name = 'myscrape'
    start_urls = ['https://indianexpress.com/section/india/']

    def parse(self, response):
        for product in response.css('div.nation'):
            yield{
                Headlines : first.css('h2.title').getall()
            }
            
    

Scrapping()

joe_bill.dollar · Accepted Answer

You should try adding the following which helps produce an output:

from scrapy.crawler import CrawlerProcess

process = CrawlerProcess(
    settings = {
        'FEED_URI':'stuff.jl',
        'FEED_FORMAT': 'jsonlines'
            }
process.crawl(Scrapping)
process.start()

Check the documentation for other exports available.

Here are the amendments to your code as you've made a few errors:

import scrapy
from scrapy.crawler import CrawlerProcess

class Scrapping(scrapy.Spider):
    name = 'myscrape'
    start_urls = ['https://indianexpress.com/section/india/']

    def parse(self, response):
        data = response.css('div.nation')
        
        for product in data:
            stuff = product.css('h2.title').getall()
            
            yield {
                'Headlines':stuff
            }
            
process = CrawlerProcess(
    settings = {
        'FEED_URI':'stack3.jl',
        'FEED_FORMAT':'jsonlines'
    }
)
process.crawl(Scrapping)
process.start()

Output:

{
    "Headlines": ["
							Centre to send teams to 10 states with high Omicron cases, low vaccination
							
", "
							PM Modi pays tributes to Vajpayee on his birth anniversary
							
", "
							Minister Harak Singh Rawat threatens to resign in fresh headache for BJP in Uttarakhand
							
", "
							VP Naidu pays tributes to Vajpayee on his birth anniversary
							
", "
							\u2018Recall noble teachings of Jesus Christ\u2019: PM Modi extends Christmas greetings
							
", "
							Centre hits out at Oppn over disrupting RS proceedings; respect commanded, not demanded, says Congress
							
", "
							Rs 150 crore seized from office of UP businessman; BJP, SP trade charges
							
", "
							Stories on upkeep of dams, mines winners in Regional category
							
", "
							Ludhiana blast burst water pipe, vital clues feared washed away
							
", "
							Death in Chhattisgarh village reshapes living with 43 elephants and crop losses
							
", "
							Covid: Rise in Delhi case count, neighbouring Haryana & UP impose night curfew
							
", "
							Cong \u2018casting aspersions\u2019 on constitutional authority: govt
							
", "
							PM chairs first meeting of committee on celebration of 150th birth anniversary of Sri Aurobindo
							
", "
							Vajpayee sought modus vivendi with China: Jaishankar
							
", "
							Third Covid wave in India may peak in early Feb: IIT-Kanpur study
							
", "
							India Covid updates: Govt says world seeing fourth surge of cases; UP, Haryana & Maharashtra impose fresh curbs
							
", "
							Brahmpura return gives Akali Dal shot in arm ahead of Punjab polls
							
", "
							50% of Omicron cases were double jabbed, masks and  surveillance must for breaking transmission chain
							
", "
							Will take call on holding UP polls next week, says CEC Sushil Chandra
							
", "
							Booked for Haridwar meet, ex-UP Shia Waqf Board chief turned Hindu, BJP advocate
							
", "
							Under political pressure, Nitish Kumar sets out on \u2018social reform\u2019 yatra
							
", "
							Prohibition won\u2019t be relaxed for those coming to Bihar from elsewhere: Nitish Kumar
							
", "
							Rahul placates Rawat: he is leader of Congress campaign, not CM face yet
							
", "
							Asaduddin Owaisi says video of him edited to mislead, to \u2018distract from Haridwar meet\u2019
							
", "
							Two more RSS workers arrested for SDPI leader\u2019s killing in Kerala
							"]
}

Results are not visible when using scrapy with python

Answers (1)

Related Questions