CODEWITHSUNDEEP
web-scraping
madhav mundra
madhav mundra

Reputation: 25

Data extraction through webscraping

Can someone please guide me as to how I can go about extracting data from this particular table? I have tried it multiple times but have not succeeded in extracting the required data. 

`import requests 
from bs4 import BeautifulSoup as bs
import pandas as pd
r = requests.get('https://etfdb.com/etf/ICLN/#fact-sheet', proxies = proxy_support).text
soup = bs(r,'html.parser')
da = soup.find_all('ul', {'class':'list-unstyled'})[0]
n_rows = 0
n_columns = 0
column_names = []

for row in da.find_all('li'):
td_tags = row.find('span')
if len(td_tags) > 0:
   n_rows+=1
   if n_columns == 0:
      n_columns = len(td_tags)

th_tags = row.find_all('a href') 
if len(th_tags) > 0 and len(column_names) == 0:
   for th in th_tags:
    column_names.append(th.get_text())

if len(column_names) > 0 and len(column_names) != n_columns:
            raise Exception("Column titles do not match the number of columns")


 columns = column_names if len(column_names) > 0 else range(0,n_columns)

 df = pd.DataFrame(columns = columns, index= range(0,n_rows))

 row_marker = 0

 for row in da.find_all('li'):
 column_marker = 0
 columns = row.find_all('span')
  for column in columns:
    df.iat[row_marker,column_marker] = columns.get_text()
    column_marker += 1
 if len(columns) > 0:
   row_marker += 1

For the code above I get the following error :

AttributeError: ResultSet object has no attribute 'get_text'. You're probably treating a list of items like a single item. Did you call find_all() when you meant to call find()?

Can any one tell me what I am doing wrong?

Upvotes: 0

Views: 69

Answers (1)

QHarr
QHarr

Reputation: 84465

With bs4 4.7.1. to get first table

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('https://etfdb.com/etf/ICLN/#fact-sheet')
soup = bs(r.content, 'lxml')
items = soup.select('h3:contains(Vitals) + ul li')

for item in items:
    print([i.text for i in item.select('span')])

Earlier bs versions

items = soup.select_one('h3 + ul').select('li')

for item in items:
    print([i.text for i in item.select('span')])

Upvotes: 2

Related Questions