CODEWITHSUNDEEP
pythonscrapyweb-crawler
VioGeo
VioGeo

Reputation: 117

Scrapy stops scraping but continues to crawl

I’m trying to scrape different information from several pages of a website. Until the sixteenth page, everything works: the pages are crawled, scraped and the information stock in my database, however after the sixteenth page, it stops scraping but continues to crawl. I checked the website and there are more of 470 pages with information. The HTML tags are the same, so I don't understand why it stopped scraping.

Python:

def url_lister():
    url_list = []
    page_count = 1
    while page_count < 480:
        url = 'https://www.active.com/running?page=%s' %page_count 
        url_list.append(url)
        page_count += 1 
    return url_list

class ListeCourse_level1(scrapy.Spider):
    name = 'ListeCAP_ACTIVE' 
    allowed_domains = ['www.active.com'] 
    start_urls = url_lister()
    
    def parse(self, response):    
        selector = Selector(response)
        for uneCourse in response.xpath('//*[@id="lpf-tabs2-a"]/article/div/div/div/a[@itemprop="url"]'): 
            loader = ItemLoader(ActiveItem(), selector=uneCourse)
            loader.add_xpath('nom_evenement', './/div[2]/div/h5[@itemprop="name"]/text()')
        loader.default_input_processor = MapCompose(string) 
        loader.default_output_processor = Join()
        yield loader.load_item()
    pass

The shell:

>     2018-01-23 17:22:29 [scrapy.core.scraper] DEBUG: Scraped from <200     
>     https://www.active.com/running?page=15>
>     {
>      'nom_evenement': 'Enniscrone 10k run & 5k run/walk',
>      }
>     2018-01-23 17:22:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.active.com/running?page=16> (referer: None)
>     --------------------------------------------------
>                     SCRAPING DES ELEMENTS EVENTS
>     --------------------------------------------------
>     2018-01-23 17:22:34 [scrapy.extensions.logstats] INFO: Crawled 17 pages (at 17 pages/min), scraped 155 items (at 155 items/min)
>     2018-01-23 17:22:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.active.com/running?page=17> (referer: None)
> 
> --------------------------------------------------
>                 SCRAPING DES ELEMENTS EVENTS
> -------------------------------------------------- 2018-01-23 17:22:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET
> https://www.active.com/running?page=18> (referer: None)
> --------------------------------------------------
>                 SCRAPING DES ELEMENTS EVENTS
> -------------------------------------------------- 2018-01-23 17:22:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET
> https://www.active.com/running?page=19> (referer: None)

Upvotes: 4

Views: 1558

Answers (1)

Tom&#225;š Linhart
Tom&#225;š Linhart

Reputation: 10210

This is probably caused by the fact that there are only 17 pages with the content you are looking for, while you instruct Scrapy to visit all 480 pages of form https://www.active.com/running?page=NNN. A better approach is to check on each page you visit that there is a next page and only in that case yield Request to the next page.

So, I would refactor your code to something like (not tested):

class ListeCourse_level1(scrapy.Spider):
    name = 'ListeCAP_ACTIVE' 
    allowed_domains = ['www.active.com'] 
    base_url = 'https://www.active.com/running'
    start_urls = [base_url]

    def parse(self, response):    
        selector = Selector(response)
        for uneCourse in response.xpath('//*[@id="lpf-tabs2-a"]/article/div/div/div/a[@itemprop="url"]'): 
            loader = ItemLoader(ActiveItem(), selector=uneCourse)
            loader.add_xpath('nom_evenement', './/div[2]/div/h5[@itemprop="name"]/text()')
        loader.default_input_processor = MapCompose(string) 
        loader.default_output_processor = Join()
        yield loader.load_item()
        # check for next page link
        if response.xpath('//a[contains(@class, "next-page")]'):
            next_page = response.meta.get('page_number', 1) + 1
            next_page_url = '{}?page={}'.format(base_url, next_page)
            yield scrapy.Request(next_page_url, callback=self.parse, meta={'page_number': next_page})

Upvotes: 4

Related Questions