CODEWITHSUNDEEP
pythonbeautifulsoup
chip
chip

Reputation: 1789

find tags except those with attributes: BeautifulSoup

In this page that I'm trying to scrape, i want to exclude those <td> that has attributes.

<td > Click here for a comprehensive area code list for Argentina </td>

I'd like to know what function/s to use to exclude this tag with attributes

My code gets all cities and area codes

from bs4 import BeautifulSoup
import urllib2
import re

url = "http://www.howtocallabroad.com/argentina"
html_page = urllib2.urlopen(url)
soup = BeautifulSoup(html_page)

areatable = soup.find('table',{'id':'codes'})
if areatable is None:
    print "areatable is None"
else:
    d = {}

    def chunks(l, n):
            return [l[i : i + n] for i in range(0, len(l), n)]

    all_td = areatable.findAll('td')
    print all_td

    li = dict(chunks([i.text for i in all_td], 2))
    print li

But when I try to print li, it throws an exception:

Traceback (most recent call last):
  File "extract_table.py", line 21, in <module>
    li = dict(chunks([i.text for i in all_td], 2))
ValueError: dictionary update sequence element #30 has length 1; 2 is required

This is what i get when i call areatable.findAll('td')

[
<td>Buenos Aires</td>,
<td>11</td>,
<td>La Rioja</td>,
<td>380</td>,
<td>Salta</td>,
<td>387</td>,
<td>Bahia Blanca</td>,
<td>291</td>,
<td>Mar del Plata</td>,
<td>223</td>,
<td>San Juan</td>,
<td>264</td>,
<td>Catamarca<br/></td>,
<td>383</td>,
<td>Mendoza</td>,
<td>261</td>,
<td>San Luis</td>,
<td>266</td>,
<td>Comodoro Rivadavia</td>,
<td>297</td>,
<td>Mercedes/Prov. B.A.</td>,
<td>2324</td>,
<td>San Nicolas</td>,
<td>336</td>,
<td>Concordia</td>,
<td>345</td>,
<td>Neuquen</td>,
<td>299</td>,
<td>San Rafael</td>,
<td>260</td>,
<td>Cordoba</td>,
<td>351</td>,
<td>Parana</td>,
<td>343</td>,
<td>Santa Fe</td>,
<td>342</td>,
<td>Corrientes</td>,
<td>379</td>,
<td>Posadas</td>,
<td>376</td>,
<td>Santiago del Estero</td>,
<td>385</td>,
<td>Formosa</td>,
<td>370</td>,
<td>Resistencia</td>,
<td>362</td>,
<td>Santo Tome</td>,
<td>3756</td>,
<td>Jesus Maria</td>,
<td>3525</td>,
<td>Rio Cuarto</td>,
<td>358</td>,
<td>Tandil</td>,
<td>249</td>,
<td>La Plata</td>,
<td>221</td>,
<td>Rosario</td>,
<td>341</td>,
<td>Trelew</td>,
<td>280</td>,
<td colspan="6" id="more"><a href="http://www.cnc.gov.ar/infotecnica/numeracion/indicativosinter.asp" target="_blank">Click here</a> for a comprehensive area code list for Argentina</td>
]

Upvotes: 2

Views: 1685

Answers (1)

TerryA
TerryA

Reputation: 60024

The problem is that the all_td is an odd length, so the chunks function doesn't work perfectly. Here is a simple lambda function which finds out if tags have no attributes, which you can use to only catch the <td>stuff</td> tags:

>>> all_td = filter(lambda x: x.attrs == {}, all_td)
# all_td now contains [<td>Buenos Aires</td>, <td>11</td>, <td>La Rioja</td>, <td>380</td>, <td>Salta</td>, <td>387</td>, <td>Bahia Blanca</td>, <td>291</td>, <td>Mar del Plata</td>, <td>223</td>, <td>San Juan</td>, <td>264</td>, <td>Catamarca<br/></td>, <td>383</td>, <td>Mendoza</td>, <td>261</td>, <td>San Luis</td>, <td>266</td>, <td>Comodoro Rivadavia</td>, <td>297</td>, <td>Mercedes/Prov. B.A.</td>, <td>2324</td>, <td>San Nicolas</td>, <td>336</td>, <td>Concordia</td>, <td>345</td>, <td>Neuquen</td>, <td>299</td>, <td>San Rafael</td>, <td>260</td>, <td>Cordoba</td>, <td>351</td>, <td>Parana</td>, <td>343</td>, <td>Santa Fe</td>, <td>342</td>, <td>Corrientes</td>, <td>379</td>, <td>Posadas</td>, <td>376</td>, <td>Santiago del Estero</td>, <td>385</td>, <td>Formosa</td>, <td>370</td>, <td>Resistencia</td>, <td>362</td>, <td>Santo Tome</td>, <td>3756</td>, <td>Jesus Maria</td>, <td>3525</td>, <td>Rio Cuarto</td>, <td>358</td>, <td>Tandil</td>, <td>249</td>, <td>La Plata</td>, <td>221</td>, <td>Rosario</td>, <td>341</td>, <td>Trelew</td>, <td>280</td>]

Simply, the lambda function will return True if the tag has no attributes. What filter() does is goes through each element in all_td, and runs the lambda function with each element. If the lambda function returns False with a given tag, it is removed from the list. A new list is returned.

Now when calling chunks, there will be an even amount of elements in the list, so no error should appear.

Upvotes: 4

Related Questions