CODEWITHSUNDEEP
pythonbeautifulsoup
jon
jon

Reputation: 6246

BeautifulSoup, a dictionary from an HTML table

I am trying to scrape table data from a website.

Here is a simple example table:

t = '<html><table>' +\
    '<tr><td class="label"> a </td> <td> 1 </td></tr>' +\
    '<tr><td class="label"> b </td> <td> 2 </td></tr>' +\
    '<tr><td class="label"> c </td> <td> 3 </td></tr>' +\
    '<tr><td class="label"> d </td> <td> 4 </td></tr>' +\
    '</table></html>'

Desired parse result is {' a ': ' 1 ', ' b ': ' 2 ', ' c ': ' 3 ', ' d ' : ' 4' }

This is my closest attempt so far:

for tr in s.findAll('tr'):
  k, v = BeautifulSoup(str(tr)).findAll('td')
  d[str(k)] = str(v)

Result is:

{'<td class="label"> a </td>': '<td> 1 </td>', '<td class="label"> d </td>': '<td> 4 </td>', '<td class="label"> b </td>': '<td> 2 </td>', '<td class="label"> c </td>': '<td> 3 </td>'}

I'm aware of the text=True parameter of findAll() but I'm not getting the expected results when I use it.

I'm using python 2.6 and BeautifulSoup3.

Upvotes: 18

Views: 38228

Answers (5)

codedmsg
codedmsg

Reputation: 1

Here is a solution that I believe is closer to the original attempt:

d={}
for tr in soup.findAll('tr'):
    key = tr.text.split()[0]
    val = tr.text.split()[1]
    d[key] = val

print(d)

Upvotes: 0

martin-martin
martin-martin

Reputation: 3564

BeautifulSoup and Python have evolved, so if someone comes here with newer versions:

Python>=3.7
BeautifulSoup>=4.7

Here's updated code that works:

# import bs4 and create your 'soup' object

table = soup.find('table')

headers = [header.text for header in table.find_all('th')]
results = [{headers[i]: cell for i, cell in enumerate(row.find_all('td'))}
           for row in table.find_all('tr')]

Upvotes: 16

Al Johri
Al Johri

Reputation: 2051

If you're scraping a table has an explicit "thead" and "tbody" such as:

<table>
    <thead>
        <tr>
            <th>Total</th>
            <th>Finished</th>
            <th>Unfinished</th>
        </tr>
    </thead>
    <tbody>
        <tr> <td>63</td> <td>33</td> <td>2</td> </tr>
        <tr> <td>69</td> <td>29</td> <td>3</td> </tr>
        <tr> <td>57</td> <td>28</td> <td>1</td> </tr>
    </tbody>
</table>

You can use the following:

headers = [header.text_content() for header in table.cssselect("thead tr th")]
results = [{headers[i]: cell.text_content() for i, cell in enumerate(row.cssselect("td"))} for row in table.cssselect("tbody tr")]

This will produce:

[
  {"Total": "63", "Finished": "33", "Unfinished": "2"},
  {"Total": "69", "Finished": "29", "Unfinished": "3"},
  {"Total": "57", "Finished": "28", "Unfinished": "1"}
]

P.S. This is using lxml.html. If you are using BeautifulSoup replace ".text_content()" with ".string" and ".cssselect" with ".findAll".

Upvotes: 8

rjfv
rjfv

Reputation: 51

You can follow the same approach as mvillaress, but improve it a bit, using List Comprehensions:

from BeautifulSoup import BeautifulSoup

t = '<html><table>' +\
    '<tr><td class="label"> a </td> <td> 1 </td></tr>' +\
    '<tr><td class="label"> b </td> <td> 2 </td></tr>' +\
    '<tr><td class="label"> c </td> <td> 3 </td></tr>' +\
    '<tr><td class="label"> d </td> <td> 4 </td></tr>' +\
    '</table></html>'

bs = BeautifulSoup(t)
tds = [row.findAll('td') for row in bs.findAll('tr')]
results = { td[0].string: td[1].string for td in tds }
print results

Upvotes: 5

mvillaress
mvillaress

Reputation: 509

Try this:

from BeautifulSoup import BeautifulSoup, Comment

t = '<html><table>' +\
    '<tr><td class="label"> a </td> <td> 1 </td></tr>' +\
    '<tr><td class="label"> b </td> <td> 2 </td></tr>' +\
    '<tr><td class="label"> c </td> <td> 3 </td></tr>' +\
    '<tr><td class="label"> d </td> <td> 4 </td></tr>' +\
    '</table></html>'

bs = BeautifulSoup(t)

results = {}
for row in bs.findAll('tr'):
    aux = row.findAll('td')
    results[aux[0].string] = aux[1].string

print results

Upvotes: 22

Related Questions