CODEWITHSUNDEEP
pythonpython-3.xseleniumweb-scrapingbeautifulsoup
PutsandCalls
PutsandCalls

Reputation: 1011

Unable to find div class on page

I am unable to find the div class in the following code.

The url is the following:

https://www.oddsportal.com/basketball/usa/nba/los-angeles-lakers-miami-heat-IqLamQfL/#over-under;1

The code I have is as follows:

html_doc = urllib.request.urlopen(new_url).read()
odds_soup = BeautifulSoup(html_doc, 'html.parser')
table_header = odds_soup.find_all('div', {'id' : "odds-data-table", 'class' : 'bt-2'})
list = []
table_containers = []
for tag in table_header:
    table_containers += tag.find_all('div', {'class' : 'table-container'})

But the code only returns an empty list for table_containers. I am not sure why, and would really appreciate some help.

On inspection of the website, it looks like this:

web page source

Upvotes: 0

Views: 238

Answers (3)

Ice Bear
Ice Bear

Reputation: 4086

The reason why you cannot find it cause your table_header doesn't find anything and the reason you're not getting anything from your table_header is because you're having a 404 status code. You can check your status code by just doing a .status_code and print it.

Source Wikipedia

The HTTP 404, 404 Not Found, 404, 404 Error, Page Not Found, File Not Found, or Server Not Found error message is a Hypertext Transfer Protocol (HTTP) standard response code, in computer network communications, to indicate that the browser was able to communicate with a given server, but the server could not find what ...

I made some modifications on your code and printed out the status code and it says 404. For the solution on why it is getting a 404 status code You might want to take a look at this answer or probably you can just use selenium as one of the answers. Good luck!

import requests
from bs4 import BeautifulSoup

link = "https://www.oddsportal.com/basketball/usa/nba/los-angeles-lakers-miami-heat-IqLamQfL/#over-under;1"

html_doc = requests.get(link)
print(html_doc.status_code)
odds_soup = BeautifulSoup(html_doc.content, 'html5lib')

table_header = odds_soup.find('div',{"id":"odds-data-table"})


'''
list = []
table_containers = []
for tag in table_header:
    table_containers += tag.find_all('div', {'class' : 'table-container'})
'''

Output:

404
[Finished in 2.1s]

Upvotes: 1

Arundeep Chohan
Arundeep Chohan

Reputation: 9969

Need to use selenium for this and pass to BS4. Then .append() to your list or print it.

driver = webdriver.Chrome()
driver.get('https://www.oddsportal.com/basketball/usa/nba/los-angeles-lakers-miami-heat-IqLamQfL/#over-under;1')
odds_soup = BeautifulSoup(driver.page_source , 'html.parser')
table_header = odds_soup.find_all("div", class_="table-header-light odd first")
for tag in table_header:
    print(tag)

Displays

<div class="table-header-light odd first"><strong><a href="" onclick="page.togleTableContent('P-201.50-0-0',this);return false;">Over/Under +201.5 </a></strong><span class="avg chunk-odd-payout"></span><span class="avg chunk-odd nowrp"></span><span class="avg chunk-odd nowrp"></span><span class="odds-cnt">(0)</span><span class="odds-co"><a class="more" href="" onclick="page.togleTableContent('P-201.50-0-0',this);return false;">Compare odds</a></span></div>

Upvotes: 0

wasif
wasif

Reputation: 15508

Just select them like:

soup.find_all("div", class_="table-header-light odd first")

Upvotes: 0

Related Questions