CODEWITHSUNDEEP
pythonhtmlbeautifulsoupextract
Mark K
Mark K

Reputation: 9376

Python BeautifulSoup to extract table from saved HTML webpages

I'm trying to extract data from saved HTML webpages using Python 2.7 + Windows.

There are multiple saved HTML webpages, which are similar and each contains a table of 5 columns. The number of rows is not fixed.

The source code looks like:

text = '''

<table id="MainTable" class="KTable" cellspacing="0" cellpadding="0" border="0">
            <tbody><tr>
                <td class="KGI"></td><td></td><td></td><td></td><td></td><td></td>
            </tr><tr id="ASPxJ1_DXGroupRowExp0" class="KGroupRow">
                <td class="K"><img class="dxJ_gvExpandedButton" src="/DXR.axd?r=1_19-RP" alt="[Collapse]"></td><td class="K" colspan="5">Supplier Code  (Count=6, <span class="grid_sumlabel">Record Count:</span><span class="grid_sumdata">86</span>) (next page)</td>
            </tr><tr id="Row1" class="row_data">
                <td class="IndentD">Â&nbsp;</td><td class="apv"><a class="dxeHyperlink" href="admin.aspx">3617</a></td><td class="class0">German</td><td class="apv">EU</td><td class="apv" align="right">2012</td><td class="apv" align="right">2013</td>
            </tr><tr id="Row2" class="row_data row_dataAlt aspxgridAltStyle">
                <td class="IndentD">Â&nbsp;</td><td class="apv"><a class="dxeHyperlink" href="admin.aspx">3617</a></td><td class="class0"><a href="ad.aspx">Belgium</a></td><td class="apv">EU</td><td class="apv" align="right">2014</td><td class="apv" align="right">2015</td>
            </tr><tr id="Row3" class="row_data">
                                                        …
                                                        …
                                                        …
                                                        …
                                                        …
            </tr><tr id="Row50" class="row_data">
            </tr>
        </tbody></table>


'''

What I want is to take the table contents and put/save them in .xls files.

What I do is:

soup = BeautifulSoup(text)
aa = soup.find_all('table')[0].tbody.find_all('tr')

for a in aa:
    print a.text

it gives all the contents but all in 1 line.

I tried:

aa = soup.find_all(id = 'MainTable')

for a in aa:
    for b in a.find_all(id = 'Row2'):
        print b.text

it gives the contents of specific row but still in 1 line.

3617BelgiumEU20142015

This isn't sufficient and uncertain rows in the html file is a problem too.

What I want is the “3617”, “Belgium”, “EU”, “2014” and “2015” separately so that I can save them in .xls files.

What’s the best way to extract the table nicely?

Upvotes: 1

Views: 270

Answers (3)

changhwan
changhwan

Reputation: 1112

find_all() can search all texts using parameter text=True

e.find_all(text=True)

On your code

aa = soup.find_all(id = 'MainTable')
for a in aa:
    for b in a.find_all(id = 'Row2'):
        print b.find_all(text=True)

will print below:

[u'\n', u'\xc2\xa0', u'3617', u'Belgium', u'EU', u'2014', u'2015', u'\n']

Note that \n is also text so you should modify your data to not have \n or filter it out.

Upvotes: 0

Anand S Kumar
Anand S Kumar

Reputation: 91009

In your second try, you can try find_all to get all <td> inside each row and then loop over them printing their text -

>>> aa = soup.find_all(id = 'MainTable')
>>> for a in aa:
...     for b in a.find_all(id = 'Row2'):
...        for c in b.find_all('td'):
...           print(c.text)
...
A 
3617
Belgium
EU
2014
2015

Upvotes: 0

alecxe
alecxe

Reputation: 474221

You should iterate over td elements for each row:

for row in soup.select("table#MainTable tr[id^=Row]"):
    for cell in row.find_all("td"):
        print cell.text

Note that I'm using a CSS selector to locate the table rows.

Upvotes: 2

Related Questions