CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoupfindall
ak27
ak27

Reputation: 13

beautifulsoup webscraper problem: can't find tables on webpage

I want to get tables from this website with this code:

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = 'https://www.flashscore.pl/pilka-nozna/'
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")
containers = page_soup.find_all('table', {'class': 'soccer'})

print(len(containers))

But when I try to check how much tables I get by print(len(containers)), I get 0. Any solutions?

edit: image of contained tables

Upvotes: 1

Views: 313

Answers (1)

chitown88
chitown88

Reputation: 28595

it's possible the page is dynamic. you can use requests-html which allows you to let the page render before pulling the html, or you can use Selenium, as I did here.

This resulted in 42 elements of table class="soccer"

import bs4 
from selenium import webdriver 

url = 'https://www.flashscore.pl/pilka-nozna/'

browser = webdriver.Chrome('C:\chromedriver_win32\chromedriver.exe')
browser.get(url)

html = browser.page_source
soup = bs4.BeautifulSoup(html,'html.parser')  

containers = soup.find_all('table', {'class': 'soccer'})

browser.close()



In  [11]: print(len(containers))
42

Upvotes: 4

Related Questions