CODEWITHSUNDEEP
htmlweb-scrapingbeautifulsoup
Yossi Bondarev
Yossi Bondarev

Reputation: 1

Beautiful Soup cannot find class which exists on web page

I am trying to scrape https://www.espn.com/nba/scoreboard There is a sort of date object there which contains links to box scores of previously played games.

I have tried :

divs = soup.find('div',class_='datepicker datepicker-dropdown datepicker-orient-left datepicker-orient-top')

print(divs)

output is: None

Other classes can be found, but for some reason this class cannot. I think it has something to do with the fact that this class is related to a calendar which pops up within the web page (without changing the url).

Upvotes: 0

Views: 50

Answers (2)

Ed Ward
Ed Ward

Reputation: 2331

This seems to work:

import bs4

html = """<div class="datepicker datepicker-dropdown datepicker-orient-left datepicker-orient-top" style="top: 35px; left: 0px; display: block;">"""

soup = bs4.BeautifulSoup(html, "html.parser")

class_ = "datepicker datepicker-dropdown datepicker-orient-left datepicker-orient-top".split(" ")
divs = [div for div in soup.find_all("div") if all(x in div.attrs["class"] for x in class_)]

print(divs)

Output:

[<div class="datepicker datepicker-dropdown datepicker-orient-left datepicker-orient-top" style="top: 35px; left: 0px; display: block;"></div>]

Upvotes: 0

Pavlo Zin
Pavlo Zin

Reputation: 770

It is possible that some portion of HTML is added by .js script dynamically after the page is loaded.

If this is the case you won't be able to load it using soup...

Upvotes: 1

Related Questions