CODEWITHSUNDEEP
python-3.xweb-scrapingbeautifulsoup
Silvia Islam Eera
Silvia Islam Eera

Reputation: 61

How to get all the p tags from the soup object

I was scraping reviews of Kickstarter. I was able to get the name of the reviewer and the 1st p tag that contains the review. But for some reviews there's additional p tags that I want to scrape. How to do that.

Here is the code I've written.

from selenium import webdriver
from bs4 import BeautifulSoup as bs
import requests

driver = webdriver.Chrome(
    executable_path=r'C:\Users\silvi\projects\selenium_basics\chromedriver.exe')
url = 'https://www.sitejabber.com/reviews/kickstarter.com'
# driver.get(url)

soup = bs(requests.get(url).content, "html.parser")

for review in soup.select('div .url-reviews .review'):
    name = review.a.get_text(strip=True, separator=" ")
    customer_review = review.find('p').text
    print(name)
    print(customer_review)
    print()
driver.quit()

Upvotes: 0

Views: 74

Answers (2)

virxen
virxen

Reputation: 438

from selenium import webdriver
from bs4 import BeautifulSoup as bs
import requests

driver = webdriver.Chrome(
    executable_path=r'C:\Users\silvi\projects\selenium_basics\chromedriver.exe')
url = 'https://www.sitejabber.com/reviews/kickstarter.com'
# driver.get(url)

soup = bs(requests.get(url).content, "html.parser")
divs=soup.findAll('div',{"class": ["url-reviews","review"]})
for review in divs:
    name = review.a.get_text(strip=True, separator=" ")
    print(name)
    customer_review = review.find_all('p')
    print("========================")
    for p in customer_review:
        print(p.text)
        print(" - - - -")
driver.quit()

use customer_review = review.find_all('p') and with a for print its elements text

Upvotes: 0

Luke Storry
Luke Storry

Reputation: 6722

You can use review.find_all('p'), then use a list comprehension to get the text out of all of them, skipping the first as you already have that in customer_review

for review in soup.select('div .url-reviews .review'):
    name = review.a.get_text(strip=True, separator=" ")
    customer_review = review.find('p').text
    other_ps = [p.text for p in review.find_all('p')[1:]]
    
    print(name)
    print(customer_review)
    print(other_ps)

Output: enter image description here

Upvotes: 1

Related Questions