CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Michael Geduldig
Michael Geduldig

Reputation: 11

Python BeautifulSoup scraping tables from a webpage

I am trying to scrape data from a webpage that has a table of users that are currently signed into the site

I'm using the below code to sign into the site

browser = RoboBrowser()
loginURL = 'https://geico.aisreview.com/ais/admin.aspx'
browser.open(loginURL)
form = browser.get_form(id='form1')
form['txtPWD'].value = 'myPassword'
browser.submit_form(form)

And I'm using this code to try and pull the data from the table. Right now I'm just trying to print it for testing purposes

soup = BeautifulSoup(loginURL)
table = soup.find_all("table", {"class": "rgMasterTable"})
for myTable in table:
  table_body = myTable.find('tbody')
  try:
    rows = table_body.find_all('tr')
    for tr in rows:
      cols = tr.find_all('td')
      for td in cols:
        print td.text
  except:
    print "no tbody found"

When running the code I don't get any errors, but nothing prints out. I was able to determine that the for loop is never being entered but I can't figure out why.

Upvotes: 0

Views: 1375

Answers (1)

Pyd
Pyd

Reputation: 6159

you can use pandas.read_html to read tables from html

import pandas as pd
import requests

loginURL='http://example.com'
res=requests.get(loginURL)

tables=pd.read_html(res.text) # return list of tables
print(tables)#will display all the tables, please slice the list for your required table.

or you can directly give the url like pd.read_html(loginURL)

Upvotes: 1

Related Questions