CODEWITHSUNDEEP
pythonscrapy
user46257
user46257

Reputation: 129

make scrapy move to next page recursively

I'm trying to scrape this page using scrapy. I can succesfully scrape the data on the page, but I want to be able to scrape data from the other pages too. (the ones that say next). heres the relevant part of my code:

def parse(self, response):
    item = TimemagItem()
    item['title']= response.xpath('//div[@class="text"]').extract()
    links = response.xpath('//h3/a').extract()
    crawledLinks=[]
    linkPattern = re.compile("^(?:ftp|http|https):\/\/(?:[\w\.\-\+]+:{0,1}[\w\.\-\+]*@)?(?:[a-z0-9\-\.]+)(?::[0-9]+)?(?:\/|\/(?:[\w#!:\.\?\+=&%@!\-\/\(\)]+)|\?(?:[\w#!:\.\?\+=&%@!\-\/\(\)]+))?$")

    for link in links:
        if linkPattern.match(link) and not link in crawledLinks:
            crawledLinks.append(link)
        yield Request(link, self.parse)

    yield item

I'm getting the right information: the titles from the linked pages, but it simply isn't 'navigating'. how do I tell scrapy to navigate?

Upvotes: 3

Views: 2917

Answers (1)

André Teixeira
André Teixeira

Reputation: 2562

Take a look on Scrapy Link Extractors documentation. They are the correct way to tell your spider to follow the links on the page.

Taking a look on the page you want to crawl, I believe you should make it with 2 extractor rules. Here is an example of a simple spider with rules that fit on your TIMES web page needs:

from scrapy.contrib.spiders import CrawlSpider,Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

class TIMESpider(CrawlSpider):
    name = "time_spider"
    allowed_domains = ["time.com"]
    start_urls = [
        'http://search.time.com/results.html?N=45&Ns=p_date_range|1&Ntt=&Nf=p_date_range%7cBTWN+19500101+19500130'
    ]

    rules = (
        Rule (SgmlLinkExtractor(restrict_xpaths=('//div[@class="tout"]/h3/a',))
            , callback='parse'),
        Rule (SgmlLinkExtractor(restrict_xpaths=('//a[@title="Next"]',))
            , follow= True),
        ) 

    def parse(self, response):
        item = TimemagItem()
        item['title']= response.xpath('.//title/text()').extract()

        return item

Upvotes: 3

Related Questions