CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Abhijeet Pal
Abhijeet Pal

Reputation: 468

Scrapy is not Crawling the next page url

My spider is not crawling the page 2 but the XPath is returning the correct next page link which is an absolute link to next page.

Here is my code 

from scrapy import Spider
from scrapy.http import Request, FormRequest



class MintSpiderSpider(Spider):

    name = 'Mint_spider'
    allowed_domains = ['example.com']
    start_urls = ['http://www.example.com/']

    def parse(self, response):
        urls =  response.xpath('//div[@class = "post-inner post-hover"]/h2/a/@href').extract()

        for url in urls:
            yield Request(url, callback=self.parse_lyrics)

        next_page_url = response.xpath('//li[@class="next right"]/a/@href').extract_first()
        if next_page_url:
            yield scrapy.Request(next_page_url, callback=self.parse)


    def parse_foo(self, response):
        info = response.xpath('//*[@class="songinfo"]/p/text()').extract()
        name =  response.xpath('//*[@id="lyric"]/h2/text()').extract()

        yield{
            'name' : name,
            'info': info
        }

Upvotes: 2

Views: 2833

Answers (1)

Adrien Blanquer
Adrien Blanquer

Reputation: 2061

The problem is that next_page_url is a list, and it needs to be an url as a string. You need to use the extract_first() function instead of extract() in next_page_url = response.xpath('//li[@class="next right"]/a/@href').extract().

UPDATE

You have to import scrapy since you are using yield scrapy.Request(next_page_url, callback=self.parse)

Upvotes: 5

Related Questions