CODEWITHSUNDEEP
pythonweb-scrapingscrapy
emon
emon

Reputation: 35

Scrapy results are repeating

I am trying to get names of the songs from this site https://pagalworld.me/category/11598/Latest%20Bollywood%20Hindi%20Mp3%20Songs%20-%202017.html using link extractor but the results are repeating. 

import scrapy
from scrapy import Request
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
class RedditSpider(CrawlSpider):
    name='pagalworld'
    allowed_domains = ["pagalworld.me"]
    start_urls=['https://pagalworld.me/category/11598/Latest%20Bollywood%20Hindi%20Mp3%20Songs%20-%202017.html']
    rules = ( 
        Rule(
        LinkExtractor(restrict_xpaths='//div/ul'), 
        follow=True, 
        callback='parse_start_url'),
    )  
    def parse_start_url(self, response):
        songName= response.xpath('//li/b/a/text()').extract()

        for item in songName:

            yield {"songName":item,
        "URL":resposne}

Output

Upvotes: 1

Views: 378

Answers (1)

Granitosaurus
Granitosaurus

Reputation: 21406

Everything seems to be correct with your spider. However if you look at the song page it offers two versions of each song:

$ scrapy shell "https://pagalworld.me/files/12450/Babumoshai%20Bandookbaaz%20(2017)%20Movie%20Mp3%20Songs.html"
>[1]: response.xpath('//li/b/a/text()').extract()
<[1]: 
['03 Aye Saiyan - Babumoshai Bandookbaaz 190Kbps.mp3',
 '03 Aye Saiyan - Babumoshai Bandookbaaz 320Kbps.mp3',
 '01 Barfani - Male (Armaan Malik) 190Kbps.mp3',
 '01 Barfani - Male (Armaan Malik) 320Kbps.mp3',
 '02 Barfani - Female (Orunima Bhattacharya) 190Kbps.mp3',
 '02 Barfani - Female (Orunima Bhattacharya) 320Kbps.mp3']

One version is lower 190kbps quality and the other is higher 320kbps quality.
In this you probably want just to keep one of those:

>[2]: response.xpath('//li/b/a/text()[contains(.,"320Kb")]').extract()
<[2]: 
['03 Aye Saiyan - Babumoshai Bandookbaaz 320Kbps.mp3',
 '01 Barfani - Male (Armaan Malik) 320Kbps.mp3',
 '02 Barfani - Female (Orunima Bhattacharya) 320Kbps.mp3']

Edit: Seems like there are also duplication issues. Try disabling follow=True on your link extractor since in this case you don't want to follow.

Upvotes: 1

Related Questions