CODEWITHSUNDEEP
pythonselenium-webdriverbeautifulsoup
user2293224
user2293224

Reputation: 2220

Python: Loading all web content in selenium

I am trying to retrieve all reviewers comments for a particular app (https://play.google.com/store/apps/details?id=com.getsomeheadspace.android&hl=en&showAllReviews=true) using selenium and beautifulsoup. I load the link by using following code:

driver = webdriver.Chrome(path)
driver.get('https://play.google.com/store/apps/details?id=com.tudasoft.android.BeMakeup&hl=en&showAllReviews=true')

The above command does not load all reviewers comments. I mean it only loads the first 39 comments and does not load remaining comments. Is there any way to load all comments in single go?

Upvotes: 0

Views: 583

Answers (2)

KunduK
KunduK

Reputation: 33384

You can use infinite loop and load the page until the Show More element is found because of lazy loading.To slowdown the loop I have used time.sleep(1). It gives 200 reviews on that page.If you want to get more then you need to click on Show More again.

However some the review format is not supporting hence try..except block.Hope this will helps.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get('https://play.google.com/store/apps/details?id=com.tudasoft.android.BeMakeup&hl=en&showAllReviews=true')

while True:
  driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
  time.sleep(1)
  elements=WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, 'div.UD7Dzf')))
  if len(driver.find_elements_by_xpath("//span[text()='Show More']"))>0:
      break;

print(len(elements))
allreview=[]
try:
   for review in elements:
       allreview.append(review.text)
except:
    allreview.append("format incorrect")

print(allreview)

Upvotes: 1

Bendik Knapstad
Bendik Knapstad

Reputation: 1458

Looks like you have to scroll down to get all the information on the page.

try this:

driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")

You may have to do that a couple of times to load all the data

Upvotes: 0

Related Questions