CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
0004
0004

Reputation: 1260

BS4 - Web Scraping - Searching div. class =

I am trying to scrape '95% liked this Movie' from the image below, but getting no result when I specify the tag via the class - view code below. Any idea on how I can attain this?

import bs4, requests
from bs4 import BeautifulSoup
res = requests.get('https://www.google.com/search?rlz=1C5CHFA_enUS879US879&sxsrf=ALeKk00cw9xBpC8OWgCnKhMSIGOi4xb3sw%3A1590372307467&ei=0yfLXrSQHNHa9AOzh6jIAg&q=titanic+google+play&oq=Titanic+&gs_lcp=CgZwc3ktYWIQAxgAMgQIIxAnMgoIABCDARAUEIcCMgcIABCDARBDMgUIABCRAjIFCAAQkQIyBwgAEIMBEEMyBAgAEEMyBwgAEIMBEEMyBAgAEEMyBAgAEEM6BAgAEEc6AggAOgUIABCDAVCcLFjMOmCEQ2gBcAN4AIABbIgBigaSAQM4LjGYAQCgAQGqAQdnd3Mtd2l6&sclient=psy-ab')

res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'html.parser')
news = [p.text.strip() for p in soup.select('h1 ~ p') if p.find('font')]

soup = BeautifulSoup(res.content, 'html.parser')
content = BeautifulSoup(res.content, 'html.parser')
content.find_all(class="srBp4.Vrkhme")`

![enter image description here]1

Upvotes: 0

Views: 1437

Answers (1)

furas
furas

Reputation: 142651

Google uses JavaScript to display results - to get 95% ... I had to use Selenium to control real web browser which can run JavaScript. And I had to use query titanic movie instead of titanic google play

import selenium.webdriver

url = 'https://www.google.com/search?q=titanic+movie'

#driver = selenium.webdriver.Chrome()
driver = selenium.webdriver.Firefox()
driver.get(url)

item = driver.find_element_by_class_name('srBp4.Vrkhme')
print(item.text.strip())

EDIT: I get it also with requests/BeautifulSoup but I had to use full header User-Agent. It doesn't work with short Mozilla/5.0

And it needs class "srBp4 Vrkhme" without dot. And it has to be class_= with _

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'}
r = requests.get('https://www.google.com/search?q=titanic+movie', headers=headers)

soup = BeautifulSoup(r.content, 'html.parser')
item = soup.find('div', class_="srBp4 Vrkhme")
print(item.get_text(strip=True, separator=' '))

Upvotes: 1

Related Questions