CODEWITHSUNDEEP
pythonweb-scrapinghtml-tablebeautifulsouppython-requests
RRUDARY
RRUDARY

Reputation: 71

Scraping Each Row from an HTML Table

I'm scraping an HTML table from a webpage, but it's just pulling the contents of the first row over and over as opposed to the unique values in each row. It seems like the positional arguments (tds[0]-tds[5]) only apply to the first row, I just don't know how to instruct the code to move on to each new row.

import requests
from bs4 import BeautifulSoup



headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}


url = 'https://www.fdic.gov/bank/individual/failed/banklist.html'
r = requests.get(url, headers = headers)

soup = BeautifulSoup(r.text, 'html.parser')


mylist5 = []
for tr in soup.find_all('table'):
    tds = tr.findAll('td')
    for x in tds:
        output5 = ("Bank: %s, City: %s, State: %s, Closing Date: %s, Cert #: %s, Acquiring Inst: %s \r\n" % (tds[0].text, tds[1].text, tds[2].text, tds[5].text, tds[3].text, tds[4].text))
        mylist5.append(output5)
        print(output5)

Upvotes: 0

Views: 47

Answers (3)

QHarr
QHarr

Reputation: 84465

I personally would use pandas here:

import pandas as pd

table = pd.read_html('https://www.fdic.gov/bank/individual/failed/banklist.html')[0]
print(table)

Upvotes: 0

Ajax1234
Ajax1234

Reputation: 71451

You can use find_all with a list comprehension:

import requests
from bs4 import BeautifulSoup as soup
d = soup(requests.get('https://www.fdic.gov/bank/individual/failed/banklist.html').text, 'html.parser')
h, data = [i.text for i in d.find_all('th')], [[i.text for i in b.find_all('td')] for b in d.find_all('tr')[1:]]

Output (shortened, due to SO's character limit):

['Bank Name', 'City', 'ST', 'CERT', 'Acquiring Institution', 'Closing Date', 'Updated Date']
[['The Enloe State Bank', 'Cooper', 'TX', '10716', 'Legend Bank, N. A.', 'May 31, 2019', 'June 5, 2019'], ['Washington Federal Bank for Savings', 'Chicago', 'IL', '30570', 'Royal Savings Bank', 'December 15, 2017', 'February 1, 2019'], ['The Farmers and Merchants State Bank of Argonia', 'Argonia', 'KS', '17719', 'Conway Bank', 'October 13, 2017', 'February 21, 2018'], ['Fayette County Bank', 'Saint Elmo', 'IL', '1802', 'United Fidelity Bank, fsb', 'May 26, 2017', 'January 29, 2019'], ['Guaranty Bank, (d/b/a BestBank in Georgia & Michigan) ', 'Milwaukee', 'WI', '30003', 'First-Citizens Bank & Trust Company', 'May 5, 2017', 'March 22, 2018'], ['First NBC Bank', 'New Orleans', 'LA', '58302', 'Whitney Bank', 'April 28, 2017', 'January 29, 2019'], ['Proficio Bank', 'Cottonwood Heights', 'UT', '35495', 'Cache Valley Bank', 'March 3, 2017', 'January 29, 2019'], ]

Upvotes: 0

Andrej Kesely
Andrej Kesely

Reputation: 195418

I slightly modified your code - I'm ignoring the first row (header) and then iterating by rows (tr), not just td:

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}

url = 'https://www.fdic.gov/bank/individual/failed/banklist.html'
r = requests.get(url, headers = headers)

soup = BeautifulSoup(r.text, 'html.parser')


mylist5 = []
for tr in soup.find_all('table'):
    rows = tr.findAll('tr')[1:]
    for row in rows:
        row = row.findAll('td')
        output5 = ("Bank: %s, City: %s, State: %s, Closing Date: %s, Cert #: %s, Acquiring Inst: %s \r\n" % (row[0].text, row[1].text, row[2].text, row[5].text, row[3].text, row[4].text))
        mylist5.append(output5)
        print(output5)

Prints:

Bank: The Enloe State Bank, City: Cooper, State: TX, Closing Date: May 31, 2019, Cert #: 10716, Acquiring Inst: Legend Bank, N. A. 

Bank: Washington Federal Bank for Savings, City: Chicago, State: IL, Closing Date: December 15, 2017, Cert #: 30570, Acquiring Inst: Royal Savings Bank 

Bank: The Farmers and Merchants State Bank of Argonia, City: Argonia, State: KS, Closing Date: October 13, 2017, Cert #: 17719, Acquiring Inst: Conway Bank 

Bank: Fayette County Bank, City: Saint Elmo, State: IL, Closing Date: May 26, 2017, Cert #: 1802, Acquiring Inst: United Fidelity Bank, fsb 

Bank: Guaranty Bank, (d/b/a BestBank in Georgia & Michigan) , City: Milwaukee, State: WI, Closing Date: May 5, 2017, Cert #: 30003, Acquiring Inst: First-Citizens Bank & Trust Company 

Bank: First NBC Bank, City: New Orleans, State: LA, Closing Date: April 28, 2017, Cert #: 58302, Acquiring Inst: Whitney Bank 

Bank: Proficio Bank, City: Cottonwood Heights, State: UT, Closing Date: March 3, 2017, Cert #: 35495, Acquiring Inst: Cache Valley Bank

...etc

Upvotes: 1

Related Questions