CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouphtml-parsing
wjrhee
wjrhee

Reputation: 13

BeautifulSoup (bs4) does not find all tags

I'm using Python 3.5 and bs4

The following code will not retrieve all the tables from the specified website. The page has 14 tables but the return value of the code is 2. I have no idea what's going on. I manually inspected the HTML and can't find a reason as to why it's not working. There doesn't seem to be anything special about each table.

import bs4
import requests

link = "http://www.pro-football-reference.com/players/B/BradTo00.htm"

htmlPage = requests.get(link)
soup = bs4.BeautifulSoup(htmlPage.content, 'html.parser')
all_tables = soup.findAll('table')
print(len(all_tables))

What's going on?

EDIT: I should clarify. If I inspect the soup variable, it contains all of the tables that I expected to see. How am I not able to extract those tables from soup with the findAll method?

Upvotes: 0

Views: 1065

Answers (1)

宏杰李
宏杰李

Reputation: 12168

this page is rendered by javascript, and if you disable the javascrip in you broswer, you will notice that this page only hava two table. i recommend to use selenium for this situation.

Upvotes: 2

Related Questions