CODEWITHSUNDEEP
pythonpython-2.7web-scrapingscrapy
iampj
iampj

Reputation: 203

Scrapy Follow link

I have the following CrawlSpider which I can't get to follow links on a university website. I think this is because of the precarious markup but I am not sure. I have tried to add a rule but it won't follow. How can I make this work?

It works as a single page spider, and scrapes page 1 ok, but doesn't follow links.

Note, not homework, just me playing about and got board of scraping Dmoz. All help is appreciated.

# -*- coding: utf-8 -*-
from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

from example.items import ExampleItem

class ExampleSpider(CrawlSpider):
    name = "example"
    allowed_domains = ["example.ac.uk"]
    start_urls = (
        'http://www.example.ac.uk/courses/course-finder?query=&f.Year_of_entry|E=2015/16&f.Type|D=Undergraduate',
        ''
    )

    rules = (Rule (SgmlLinkExtractor(allow=("index\.php", ), callback="parse"),))

    def parse(self, response):
        sel = Selector(response)
        sites = sel.xpath('//div[@id="course_list"]')
        items = []

        for site in sites:
            item = ExampleItem()
            item['link'] = site.xpath('//h2/a/@href').extract()
            item['name'] = site.xpath('//h2/a/text()').extract()
            items.append(item)

        return items

The pagination markup on the website is as follows:

   <div class="pagination">
            <ul>
                <li><i class="fa fa-chevron-left"></i><span>Previous</span></li>

                    <li><span>Go to page</span> 1</li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=11"><span>Go to page</span> 2</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=21"><span>Go to page</span> 3</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=31"><span>Go to page</span> 4</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=41"><span>Go to page</span> 5</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=51"><span>Go to page</span> 6</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=61"><span>Go to page</span> 7</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=71"><span>Go to page</span> 8</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=81"><span>Go to page</span> 9</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=91"><span>Go to page</span> 10</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=11"><i class="fa fa-chevron-right"></i><span>Next</span></a></li>
            </ul>
        </div>

Upvotes: 2

Views: 773

Answers (1)

alecxe
alecxe

Reputation: 474001

At least the first problem you have is that you are defining the callback inside a link extractor, but should be defining on the rule level:

rules = (
    Rule(LinkExtractor(allow=("index\.php", )), callback="parse_result"),
)

def parse_result(self, response):
    ...

Besides, you need a separate rule to follow pagination:

rules = (
    Rule(LinkExtractor(allow=("index\.php", )), callback="parse_result"),
    Rule(LinkExtractor(restrict_xpaths='//div[@class="pagination"]'), follow=True),
)

Upvotes: 2

Related Questions