CODEWITHSUNDEEP
web-crawlerscrapy
Miguel
Miguel

Reputation: 3064

Scrapy: How to check and stop crawling

I'm crawling a list of pages, where each page has a list of urls that I need also to be parses. I'm looping over these first pages but I don't know a priori when should I stop the crawling. For example this one is still to be parsed:

http://www.cmjornal.pt/opiniao/colunistas/acacio-pereira/MoreContent?firstContent=183

but not this one not because is already empty:

http://www.cmjornal.pt/opiniao/colunistas/acacio-pereira/MoreContent?firstContent=200

So my question is: how can I stop the crawler with a condition found from a url parsing? I tried to use CloseSpider() but it doesn't work, because it completely close the spider, before the other urls are parsed.

I show the code I'm using with the CloseSpider():

class CmSpider(scrapy.Spider):
    name = "historical"
    start_urls = ['http://www.cmjornal.pt/opiniao/colunistas/acacio-pereira/MoreContent?firstContent=']
    hostname = 'http://www.cmjornal.pt'


    def parse(self, response):
        for i in range(180,200,3):
            url = response.url + str(i)
            yield scrapy.Request(url,callback=self.parse_page,priority = 1)

    def parse_page(self,response):
        if len(response.xpath('/html/body//*')) <= 2:
            raise CloseSpider('bandwidth_exceeded')
        else:
            pass


        articles_url = response.xpath('//*[@class="lead"]/../h3/a/@href').extract()
        for url in articles_url:
            url = self.hostname+url
            item = CmItem()
            item['hostname'] = self.hostname
            request =  scrapy.Request(url,callback=self.parse_article)
            request.meta['item'] = item
            yield request

    def parse_article(self,response):
        item = response.meta['item']

        (...)

Note: for this particular case I know when the content will end, but I need to run this for many other cases that I don't know such limit.

Upvotes: 1

Views: 1069

Answers (1)

Wilfredo
Wilfredo

Reputation: 1548

You should stop yielding more requests instead of closing the spider, something like this:

# -*- coding: utf-8 -*-
import scrapy
from w3lib.url import add_or_replace_parameter
from w3lib.url import url_query_parameter


class HistorialSpider(scrapy.Spider):
    name = 'historial'
    allowed_domains = ['cmjornal.pt']

    def start_requests(self):
        base_url = 'http://www.cmjornal.pt/opiniao/colunistas/acacio-pereira/MoreContent'
        new_url = add_or_replace_parameter(base_url, 'firstContent', 180)
        yield scrapy.Request(new_url, callback=self.parse_page)

    def parse_page(self, response):
        if len(response.xpath('/html/body//*')) <= 2:
           return

        next_page = int(url_query_parameter(response.url, 'firstContent')) + 1
        yield scrapy.Request(add_or_replace_parameter(response.url, 'firstContent', next_page),
                             callback=self.parse_page)


        articles_url = response.xpath('//*[@class="lead"]/../h3/a/@href').extract()
        for url in articles_url:
            yield response.follow(url, callback=self.parse_article)

    def parse_article(self, response):
        pass

Upvotes: 1

Related Questions