CODEWITHSUNDEEP
pythonscrapy
meghana V
meghana V

Reputation: 19

How to crawl all the web pages of a website. I could crawl only 2 web pages

I am crawling Website "https://www.imdb.com/title/tt4695012/reviews?ref_=tt_ql_3". The data i need is reviews and ratings from the above website. I could crawl only 2 pages. But i want reviews and ratings from all the pages of the website.

Below is the code that I have tried

I have Included multiple websites in start_urls.

class RatingSpider(Spider):
    name = "rate"
    start_urls = ["https://www.imdb.com/title/tt4695012/reviews?ref_=tt_ql_3"]
    def parse(self, response):
        ratings = response.xpath("//div[@class='ipl-ratings-bar']//span[@class='rating-other-user-rating']//span[not(contains(@class, 'point-scale'))]/text()").getall()
        texts = response.xpath("//div[@class='text show-more__control']/text()").getall()
        result_data = []
        for i in range(0, len(ratings)):
            row = {}
            row["ratings"] = int(ratings[i])
            row["review_text"] = texts[i]
            result_data.append(row)
            print(json.dumps(row))

        next_page = response.xpath("//div[@class='load-more-data']").xpath("@data-key").extract()
        next_url = response.urljoin("reviews/_ajax?ref_=undefined&paginationKey=")
        next_url = next_url + next_page[0]
        if next_page is not None and len(next_page) != 0:
            yield scrapy.Request(next_url, callback=self.parse)

Help me in Crawling all the pages of the website.

Upvotes: 0

Views: 81

Answers (1)

vezunchik
vezunchik

Reputation: 3717

You have problems with url for next_page. If you will keep starting url and use it for all the next pages, you will get all reviews data. Check this solution:

import scrapy
from urlparse import urljoin


class RatingSpider(scrapy.Spider):
    name = "rate"
    start_urls = ["https://www.imdb.com/title/tt4695012/reviews?ref_=tt_ql_3"]

    def parse(self, response):
        ratings = response.xpath("//div[@class='ipl-ratings-bar']//span[@class='rating-other-user-rating']//span[not(contains(@class, 'point-scale'))]/text()").getall()
        texts = response.xpath("//div[@class='text show-more__control']/text()").getall()
        result_data = []
        for i in range(len(ratings)):
            row = {
                "ratings": int(ratings[i]),
                "review_text": texts[i]
            }
            result_data.append(row)
            print(json.dumps(row))

        key = response.css("div.load-more-data::attr(data-key)").get()
        orig_url = response.meta.get('orig_url', response.url)
        next_url = urljoin(orig_url, "reviews/_ajax?paginationKey={}".format(key))
        if key:
            yield scrapy.Request(next_url, meta={'orig_url': orig_url})

Upvotes: 1

Related Questions