Trying to webscrape 'hellopeter.com' can't dig into the actual review

Question

I'm trying to make an automated system that will build a database of reviews on a company from hellopeter.

I want to apply sentiment analysis on this dataset i have tried using beautiful soup but i cant dig to the important information

page = requests.get('https://www.hellopeter.com/telkom/reviews/appalling-service-cdc4559e8084e0db01dd6d7e807875460607ac77-2851593')
soup = BeautifulSoup(page.content, 'html.parser')

print(soup.prettify())

the output i expected was to be able to dig down to the class where the actual review data is. what I actually get is

there is about 10 layers below the

<div id="app">
  </div>

that it outputs and i cant figure out how to get to the actual review

more specifically i want the

<p data-v-403b1c0a="" itemprop="reviewRating" itemscope="itemscope" itemtype="http://schema.org/Rating">

1

<p data-v-403b1c0a="" itemprop="reviewBody" class="full-content">

<p data-v-403b1c0a="" itemprop="name" class="is-detail-card">APPALLING SERVICE</p>

elements any advice

Answer 1

If you go to Network Tab You will get the following API.

https://api.hellopeter.com/consumer/business/telkom/reviews/appalling-service-cdc4559e8084e0db01dd6d7e807875460607ac77-2851593

Load the json first and fetch the key value of review_content

import requests
import json
page = requests.get('https://api.hellopeter.com/consumer/business/telkom/reviews/appalling-service-cdc4559e8084e0db01dd6d7e807875460607ac77-2851593')
jsondata=json.loads(page.text)
print(jsondata['review_content'])

Output:

I have had the most horrific experience with Telkom, I have never dealt with such incompetence in my whole 31 years of being alive on earth. 

We opened a contract with Telkom for unlimited WIFI in our home. Instead of Telkom opening one account they opened two and delivered two routers to our home. Why? (Only God knows) I called and notified them, they advised they would send someone out to collect the router. 

Over two weeks later, nothing was resolved! Instead of them debiting the agreed amount of R900 they debited over R3000. I have called several times to ask why this has not been canceled as yet as this was their error and all I was told is they do not know... (How sway!) The following month again I was debited over R2000. Still nothing...

Everyday when I call I am told it will be canceled within 24hours or that someone will give me a call... but NOTHING! ABSOLUTE NOTHING! I have to constantly follow up and nag and I am always getting empty promises. I have called over 20 times for the same issue to be resolved.

The staff are incompetent and the agents have so much attitude and do not care about their customers. I am up to my head in frustration. This Morning I almost broke down in tears. How do they manage to not care at all? This month I received another message to say that they will debit another amount of over R2000.

It amazes me that when it comes to clients not paying in time their services are immediately terminated yet when they have to pay you your money it becomes such a drag. Talk about double standards.

I have never in my life experienced such emotional abuse from a service provider. I started questioning to myself, is it maybe because I am black? female? I am not too sure at this point. I just do not understand how hard could it be to fix an error that they made? It surely must be a literal click of a button or something. Why is it taking so long? We are in the THIRD MONTH now. Does Telkom care about it's customers? From this experience I believe they do not.

I am sick and tired of the excuses and I would just like everything to be sorted. I have been very patient and understanding with Telkom but they are just taking advantage now.  

Answer 2

You can extract the review text using only Selenium inducing WebDriverWait for the visibility_of_element_located() and you can use the following Locator Strategy:

• Using XPATH:

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//p[@itemprop='reviewRating']//following::p[@class='full-content' and @itemprop='reviewBody']"))).get_attribute("innerHTML"))
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• Console Output:

  I have had the most horrific experience with Telkom, I have never dealt with such incompetence in my whole 31 years of being alive on earth. 
  
  We opened a contract with Telkom for unlimited WIFI in our home. Instead of Telkom opening one account they opened two and delivered two routers to our home. Why? (Only God knows) I called and notified them, they advised they would send someone out to collect the router. 
  
  Over two weeks later, nothing was resolved! Instead of them debiting the agreed amount of R900 they debited over R3000. I have called several times to ask why this has not been canceled as yet as this was their error and all I was told is they do not know... (How sway!) The following month again I was debited over R2000. Still nothing...
  
  Everyday when I call I am told it will be canceled within 24hours or that someone will give me a call... but NOTHING! ABSOLUTE NOTHING! I have to constantly follow up and nag and I am always getting empty promises. I have called over 20 times for the same issue to be resolved.
  
  The staff are incompetent and the agents have so much attitude and do not care about their customers. I am up to my head in frustration. This Morning I almost broke down in tears. How do they manage to not care at all? This month I received another message to say that they will debit another amount of over R2000.
  
   It amazes me that when it comes to clients not paying in time their services are immediately terminated yet when they have to pay you your money it becomes such a drag. Talk about double standards.
  
  I have never in my life experienced such emotional abuse from a service provider. I started questioning to myself, is it maybe because I am black? female? I am not too sure at this point. I just do not understand how hard could it be to fix an error that they made? It surely must be a literal click of a button or something. Why is it taking so long? We are in the THIRD MONTH now. Does Telkom care about it's customers? From this experience I believe they do not.
  
  I am sick and tired of the excuses and I would just like everything to be sorted. I have been very patient and understanding with Telkom but they are just taking advantage now.