CODEWITHSUNDEEP
pythonhtmlweb-scrapingbeautifulsoup
Bruno
Bruno

Reputation: 497

Python Beautiful Soup cannot find table

I have a web scraping code that cannot find a table. My code looks like: 

site = 'http://etfdb.com/compare/market-cap/'
hdr = {'User-Agent': 'Mozilla/5.0'}
req = Request(site, headers=hdr)
page = urlopen(req)
soup = BeautifulSoup(page)
table = soup.find('table', {"class":"table mm-mobile-table table-striped 
table-bordered"})

The tables HTML looks like:

<table class="table mm-mobile-table table-striped table-bordered" data-
icons-prefix="fa" data-icons="{&quot;columns&quot;:&quot;fa-th&quot;}" data-
striped="true" data-toggle="table">

but for some reason my code always returns table as none. I do not know why but any help would be appreciated. Thanks.

Upvotes: 3

Views: 5907

Answers (3)

Dan-Dev
Dan-Dev

Reputation: 9440

The problem is that there is incorrect markup making large sections of the code commented out i.e 

<!-->.

the fix was to replace these elements then parse the HTML.

from urllib2 import urlopen, Request
from bs4 import BeautifulSoup
site = 'http://etfdb.com/compare/market-cap/'
hdr = {'User-Agent': 'Mozilla/5.0'}
req = Request(site, headers=hdr)
res = urlopen(req)
rawpage = res.read()
page = rawpage.replace("<!-->", "")
soup = BeautifulSoup(page, "html.parser")
table = soup.find("table", {"class":"table mm-mobile-table table-striped table-bordered"})
print (table)

Tested on Python 2.7.12

from urllib.request import urlopen, Request
from bs4 import BeautifulSoup
site = 'http://etfdb.com/compare/market-cap/'
hdr = {'User-Agent': 'Mozilla/5.0'}
req = Request(site, headers=hdr)
res = urlopen(req)
rawpage = res.read().decode("utf-8") 
page = rawpage.replace('<!-->', '')
soup = BeautifulSoup(page, "html.parser")
table = soup.find("table", {"class":"table mm-mobile-table table-striped table-bordered"})
print (table)

Tested on Python 3.5.2

Gives:

 <table class="table mm-mobile-table table-striped table-bordered" data-icons='{"columns":"fa-th"}' data-icons-prefix="fa" data-striped="true" data-toggle="table"><thead><tr><th class="show-td" data-field="symbol">Symbol</th> <th class="show-td" data-field="name">Name</th> <th class="show-td" data-field="aum">AUM</th> <th class="show-td" data-field="avg-volume">Avg Volume</th></tr></thead><tbody><tr><td class="show-td" data-th="Symbol"><a href="/etf/SPY/">SPY</a></td> <td class="show-td" data-th="Name"><a href="/etf/SPY/">SPDR S&amp;P 500 ETF</a></td> <td class="show-td" data-th="AUM">$236,737,519.17</td> <td class="show-td" data-th="Avg Volume">73,039,883</td></tr> <tr><td class="show-td" data-th="Symbol"><a href="/etf/IVV/">IVV</a></td> <td class="show-td" data-th="Name"><a href="/etf/IVV/">iShares Core S&amp;P 500 ETF</a></td> <td class="show-td" data-th="AUM">$115,791,603.10</td> <td class="show-td" data-th="Avg Volume">3,502,931</td></tr> ...

Upvotes: 4

Masood Lapeh
Masood Lapeh

Reputation: 306

I don't know maybe it's because their code is not well-structured [deliberately] and BeautifulSoup couldn't parse it. But maybe this helps you:

import re    # to extract table with regex
table_soup = BeautifulSoup( re.findall(r'<table.*>.*</table>', page.read())[0] )

Also you can investigate further with something like this:

for tag in soup.find_all():
   if tag.name == 'div':
      if str(tag).find("table-striped") > -1:
         print repr( str(tag)[:100] )

To see which div tags contain the table, if any:

'<div class="mm-header">\n<div class="container">\n<div class="row">\n<div class="col-xs-12">\n<div class'
'<div class="container">\n<div class="row">\n<div class="col-xs-12">\n<div class="mm-user-links hidden-x'
'<div class="row mm-header-content">\n<div class="col-sm-4 mm-header-logos">\n<button class="navbar-tog'
'<div class="col-sm-4 mm-header-logos">\n<button class="navbar-toggle">\n<span class="fa fa-bars"></spa'

So I guess BeautifulSoup couldn't break down the last one into child tags.

Edit:

  1. Sorry I forgot to add page.read() to get page source first, see the second line again.

  2. If you don't want to use regex, you can use a better parser rather than the default one: html5lib

First install it:

# pip install html5lib

The you can use it by:

soup = BeautifulSoup(page, 'html5lib')

Upvotes: 0

Prem
Prem

Reputation: 11985

You need to specify what exactly you want to parse

soup = BeautifulSoup(page, 'html.parser')

Then you can very well verify if it parsed properly

print(soup.prettify())

Upvotes: 0

Related Questions