CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scrapingxpath
dwk963
dwk963

Reputation: 49

Selenium: Xpath returning blank text?

So I have this url:

https://www.amazon.com/RevitaLash-Cosmetics-RevitaBrow-Advanced-Conditioner/product-reviews/B009QZCAM6/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews

and I wanted to scrape the ratings given by each review but it's not returning anything even when I tried a few xpath variations.

The xpaths do find the 10 elements with the text "x out of 5 stars" when I search it in the page with the given xpath and here is what I have so far:

from bs4 import BeautifulSoup
import requests
import csv
import os
import pandas as pd
from selenium import webdriver


chromedriver = "path to chromedriver"
driver = webdriver.Chrome(chromedriver)

url = https://www.amazon.com/RevitaLash-Cosmetics-RevitaBrow-Advanced-Conditioner/product-reviews/B009QZCAM6/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews

driver.get(url)

ratings = driver.find_elements_by_xpath('//div[@class="a-section a-spacing-none review-views celwidget"]//div[@class="a-row"]/a[@class="a-link-normal"]/i/span')
#ratings = driver.find_elements_by_xpath('/*//div[@id="cm_cr-review_list"]//i[@data-hook="review-star-rating"]/span[@class="a-icon-alt"]')
#ratings = driver.find_elements_by_xpath('/*//div[@id="cm_cr-review_list"]//i[@data-hook="review-star-rating"]/span')

rating_row = []
for rating in ratings:
    rating_row.append(rating.text)

but when I call rating_row it just ends up returning list of blank texts when I call rating_row

> rating_row
['', '', '', '', '', '', '', '', '', '']

What am I doing wrong here and how do I solve this? This result is the same for other urls that contain other reviews as well.

Upvotes: 1

Views: 435

Answers (2)

Rola
Rola

Reputation: 1936

Replace the line rating_row.append(rating.text) with rating_row.append(rating.get_attribute('innerHTML'))

Upvotes: 3

cruisepandey
cruisepandey

Reputation: 29362

You can try with the below xpath :

//i[@data-hook='review-star-rating']/span

in code :

rating_row = []
for rating in driver.find_elements(By.XPATH, "//i[@data-hook='review-star-rating']/span"):
    rating_row.append(rating.text)

I would recommend you to have driver.implicitly_wait(30) implicit wait also. You can write this at this place :

driver = webdriver.Chrome(chromedriver)
driver.implicitly_wait(30)

Read more about Implicit wait here

Upvotes: 1

Related Questions