CODEWITHSUNDEEP
pythonseleniumweb-scrapingpaginationselenium-chromedriver
krishan
krishan

Reputation: 86

how to handle pagination and scrape using selenium

guys I'm trying to scrape amazon reviews using selenium but don't know how to handle next page URL and I want to scrape using dynamic conditions, not by self counting the pages and apply static method

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

#Using chrome browser

driver=webdriver.Chrome(executable_path='./chromedriver.exe')
driver.get('https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/dp/B08Z1HHHTD/ref=sr_1_2?dchild=1&keywords=skybags&qid=1627786382&sr=8-2')

title_of_product = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "productTitle"))
    )
print(title_of_product.text)

Reviews=WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.XPATH, "//span[@class='a-size-base review-text review-text-content']/span")))

next_button =WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CLASS_NAME,"a-last"))).click()


time.sleep(10)
driver.close()

Upvotes: 0

Views: 1247

Answers (2)

Md. Fazlul Hoque
Md. Fazlul Hoque

Reputation: 16189

I don't know whether you will like the way of pagination but simple, neat and clean, few code can complete everything the simplest way where as I scrape the site using Scrapy CrawlSpider.

CODE:

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class AmazonReviewsSpider(CrawlSpider):

    name = 'reviews'

    allowed_domains = ['www.amazon.in']

    start_urls = ['https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews']

    rules = (
        Rule(LinkExtractor(restrict_xpaths='//a[@data-hook="review-title"]'), callback='parse_item', follow=False),
        Rule(LinkExtractor(restrict_xpaths='//*[@id="cm_cr-pagination_bar"]/ul/li/a'),follow=True),
        )
        
    def parse_item(self, response):
        yield{
            'Reviewer':response.xpath('//*[@class="a-profile-name"]/text()').get()
        }

OUTPUT:

{'Reviewer': 'Shaheen Khan'}
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R1PL6L4U9NYL58/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R2OBKWAHDBDKDA/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R1PL6L4U9NYL58/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Sidhesh Mardolkar'}
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R21ZXYGSCCPE5V/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R2OBKWAHDBDKDA/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Atowar Rahman'}
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R32Y55ISEX5B6P/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R19PEVFYAI50FE/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R21ZXYGSCCPE5V/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'leo'}
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R32Y55ISEX5B6P/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Asim ahmed'}
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R19PEVFYAI50FE/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Mahavir singh'}
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/RYD458HW5E42N/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/RYD458HW5E42N/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Ashish Modanwal'}
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/RUHTIOZJGQ7YX/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/RUHTIOZJGQ7YX/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'muskan'}
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R2X2WTTAWX9V2J/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R2X2WTTAWX9V2J/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Susheel Kumar'}
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R15DM4BMSG84D8/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R2TU8L168L3NFO/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R15DM4BMSG84D8/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Shashi'}
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R2TU8L168L3NFO/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Akhil'}
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R1QZDPG9S17TDN/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R1Z8DTMO2OF7PT/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R1QZDPG9S17TDN/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Siddhartha'}
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_1?ie=UTF8&reviewerType=all_reviews> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R1Z8DTMO2OF7PT/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Shravya kale'}
2021-08-01 12:33:20 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.amazon.in/gp/customer-reviews/R1XAMZ9LKHPV8A/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/RKG3L6Y5ZDMGI/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)       
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R396ADCZXCRGSB/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/RKG3L6Y5ZDMGI/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'S singh'}
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R396ADCZXCRGSB/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Shourya satyam'}
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R2USNFWP35AWMO/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R12T25TWUEVJ80/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_3?ie=UTF8&pageNumber=3&reviewerType=all_reviews> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R3L7T6GAL2W2GC/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R2USNFWP35AWMO/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'santhakumar'}
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/RA9C9WOIMZOQR/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)       
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R3NN436HHQPS5G/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R12T25TWUEVJ80/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Harsh Gupta'}
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R3L7T6GAL2W2GC/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'vishal thakare'}
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/RA9C9WOIMZOQR/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Ayushi'}
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R3NN436HHQPS5G/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'NIKET'}
2021-08-01 12:33:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/RFMWLES7SUYSR/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_3?ie=UTF8&pageNumber=3&reviewerType=all_reviews)       
2021-08-01 12:33:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R3LZD41TT5MPRN/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_3?ie=UTF8&pageNumber=3&reviewerType=all_reviews)      
2021-08-01 12:33:21 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/RFMWLES7SUYSR/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Best prodAmazon Customer'}
2021-08-01 12:33:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R1RZQAQO5T2OAX/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_3?ie=UTF8&pageNumber=3&reviewerType=all_reviews)      
2021-08-01 12:33:21 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R3LZD41TT5MPRN/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'rama krishna . y'}
2021-08-01 12:33:21 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R1RZQAQO5T2OAX/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Amit Biswas'}

Upvotes: 1

SIM
SIM

Reputation: 22440

If I understand your question in the right way, the following should do it:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get('https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/dp/B08Z1HHHTD/ref=sr_1_2?dchild=1&keywords=skybags&qid=1627786382&sr=8-2')

product_title = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "productTitle"))).text

print(product_title)

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[@data-hook='see-all-reviews-link-foot']"))).click()

while True:
    for item in WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "[data-hook='review']"))):
        reviewer = item.find_element_by_css_selector("span.a-profile-name").text
        review = ' '.join([i.text.strip() for i in item.find_elements_by_xpath(".//span[@data-hook='review-body']")])
        print(reviewer,review)

    try:
        WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//*[@data-hook='pagination-bar']//a[contains(@href,'/product-reviews/') and contains(text(),'Next page')]"))).click()
        WebDriverWait(driver, 10).until(EC.staleness_of(item))
    except Exception as e:
        break

driver.quit()

Upvotes: 2

Related Questions