CODEWITHSUNDEEP
pythonpython-requests
Kamikaze_goldfish
Kamikaze_goldfish

Reputation: 861

Requests html page differs from browser

So when I go to a page with python requests, it is rendering a different page. Obviously that is some script doing this. Is there any way to get the requests in python to render the same page so I can scrape it? I am really just starting to learn BeautifulSoup all over again.

Page I want to scrape

page = requests.get('https://www.yellowpages.com/new-orleans-la/mip/upperline-restaurant-526381149?lid=1001797484770')
soup = BeautifulSoup(page.text, 'lxml')
# I am trying to find this class 'sales-info' but it isn't in the html when rendered.
business_name = soup.find('div', class_='sale-info')
print(business_name)

Upvotes: 0

Views: 141

Answers (1)

sniperd
sniperd

Reputation: 5274

Looks like just a small type-o, sale-info should be sales-info (note the second s) try this:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.yellowpages.com/new-orleans-la/mip/upperline-restaurant-526381149?lid=1001797484770')
soup = BeautifulSoup(page.text, 'html.parser')
# I am trying to find this class 'sales-info' but it isn't in the html when rendered.
business_name = soup.find('div', class_='sales-info')
print(business_name)

results:

<div class="sales-info"><h1>Upperline Restaurant</h1></div>

Upvotes: 2

Related Questions