CODEWITHSUNDEEP
pythonscrapyweb-crawler
e1v1s
e1v1s

Reputation: 385

Best way to get follow links scrapy web crawler

So I'm trying to write a spider to continue clicking a next button on a webpage until it can't anymore (or until I add some logic to make it stop). The code below correctly gets the link to the next page but prints it only once. My question is why isn't it "following" the links that each next button leads to? 

class MyprojectSpider(scrapy.Spider):
    name = 'redditbot'
    allowed_domains = ['https://www.reddit.com/r/nfl/?count=25&after=t3_7ax8lb']
    start_urls = ['https://www.reddit.com/r/nfl/?count=25&after=t3_7ax8lb']

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        next_page = hxs.select('//div[@class="nav-buttons"]//a/@href').extract()
        if next_page:
            yield Request(next_page[1], self.parse)
            print(next_page[1])

Upvotes: 0

Views: 203

Answers (1)

codeadict
codeadict

Reputation: 2753

To go to the next page, instead of printing the link you just need to yield a scrapy.Request object like the following code:

import scrapy

class MyprojectSpider(scrapy.Spider):
    name = 'myproject'
    allowed_domains = ['reddit.com']
    start_urls = ['https://www.reddit.com/r/nfl/']

    def parse(self, response):
        posts = response.xpath('//div[@class="top-matter"]')
        for post in posts:
            # Get your data here
            title = post.xpath('p[@class="title"]/a/text()').extract()
            print(title)
            # Go to next page
            next_page = response.xpath('//span[@class="next-button"]/a/@href').extract_first()
            if next_page:
                 yield scrapy.Request(response.urljoin(next_page), callback=self.parse)

Update: Previous code was wrong, needed to use the absolute URL and also some Xpaths were wrong, this new one should work.

Hope it helps!

Upvotes: 1

Related Questions