CODEWITHSUNDEEP
python-3.xbeautifulsoup
OrangeEfficiency
OrangeEfficiency

Reputation: 47

Python Beautifulsoup4 parsing multiple tables

I have been trying to parse multiple tables from this website i.e http://www.cboe.com/strategies/vix/optionsintro/part5.aspx

Here is my code: 

import bs4 as bs
import pandas as pd
from urllib.request import Request, urlopen
req = Request('http://www.cboe.com/strategies/vix/optionsintro/part5.aspx', headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
soup = bs.BeautifulSoup(webpage,'lxml')
table = soup.findAll('table',{'class':'table  oddeven  center  padvertical  cellborders  mobile-load'})
table_rows = table.find_all('tr')
for tr in table_rows:
    td = tr.find_all('td')
    row = [i.text for i in td]
    print(row)

However, it keeps generating me with this message :

AttributeError: 'ResultSet' object has no attribute 'find_all'

Would anyone mind helping me out?

Upvotes: 1

Views: 813

Answers (2)

宏杰李
宏杰李

Reputation: 12158

table = soup.findAll('table',{'class':'table  oddeven  center  padvertical  cellborders  mobile-load'})

This will return a list of table tag, it's a list object or ResultSet, the find_all() can only be used in tag object

import bs4 as bs

from urllib.request import Request, urlopen
req = Request('http://www.cboe.com/strategies/vix/optionsintro/part5.aspx', headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
soup = bs.BeautifulSoup(webpage,'lxml')
for tr in soup.find_all('tr'):
    td = [td for td in tr.stripped_strings]
    print(td)

out:

['Bid', 'Ask']
['VIX Dec 10 Call', '6.40', '6.80']
['VIX Dec 15 Call', '2.70', '2.90']
['VIX Dec 16 Call', '2.30', '2.40']
['VIX Dec 17 Call', '1.80', '1.90']
['VIX Dec 18 Call', '1.45', '1.55']
['VIX Dec 19 Call', '1.15', '1.25']
['VIX Dec 20 Call', '0.95', '1.00']
['Bid', 'Ask']
['VIX Dec 10 Call', '9.30', '9.70']
['VIX Dec 15 Call', '4.90', '5.20']
['VIX Dec 16 Call', '4.30', '4.60']
['VIX Dec 17 Call', '3.70', '3.90']
['VIX Dec 18 Call', '3.10', '3.30']
['VIX Dec 19 Call', '2.65', '2.75']
['VIX Dec 20 Call', '2.25', '2.35']

This website only contain two tables which are what we need, so just find all tr will return the information we need.

Upvotes: 1

Stergios
Stergios

Reputation: 3196

The following code works for me:

import urllib2
from bs4 import BeautifulSoup
opener = urllib2.build_opener()
opener.addheaders = [('User-Agent', 'Mozilla/5.0')]
url = 'http://www.cboe.com/strategies/vix/optionsintro/part5.aspx'
response = opener.open(url)
soup = BeautifulSoup(response, "lxml")

tables = soup.findAll('table',{'class':'table  oddeven  center  padvertical  cellborders  mobile-load'})
for table in tables:
    table_rows = table.find_all('tr')
    for tr in table_rows:
        td = tr.find_all('td')
        row = [i.text for i in td]
        print(row)

Upvotes: 0

Related Questions