CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
user1851527
user1851527

Reputation: 27

Why doesn't BeautifulSoup scrape the entirety of the webpage and ow to get it?

I'm trying to get some data, from this webpage. All the text is visible on the page, however, when I grab it with BSoup I cant find any of the numbers for odds.

from selenium import webdriver
from urllib2 import urlopen
from bs4 import BeautifulSoup

URL = "https://www.sportsbookreview.com/betting-odds/mlb-baseball/money-line/1st-half/?date=20160601"

driver = webdriver.Chrome()
driver.get(URL)
soup = BeautifulSoup(driver.page_source,'lxml')
driver.quit()

opener = soup.findAll('span' , {'class' : 'opener'})

print opener

What's even weirder, it worked once, then stopped for no apparent reason or changes to the code. When I checked what soup scrapes from this page, the numbers for odds weren't even included.

Why don't I get all the data and what to do to get it?

Upvotes: 0

Views: 59

Answers (1)

iliya
iliya

Reputation: 531

I think you need to use Selenium to make javascript triggered then extract data you want in this case code below can help :

options = webdriver.ChromeOptions()
# also look at options
driver = webdriver.Chrome('address_of_your_driver', chrome_options=options)
driver.get("your_URL_here")

# now you've got the source
soup = BeautifulSoup(driver.page_source, "html.parser")

Upvotes: 1

Related Questions