CODEWITHSUNDEEP
pythonpandasbeautifulsoup
user131983
user131983

Reputation: 3937

AttributeError when scraping data from URL via Python

I am using the code below to try an extract the data from the table in this URL. I asked the same question here and got an Answer for it. However, despite the code from the Answer working at that time I've now come to realize that data in the code hasn't been defined and hence the code below results in the error AttributeError: 'NoneType' object has no attribute 'text' in the line final = [[t.th.text] + [ele.text for ele in t.find_all("td")] for t in h[-1].find_all_next("tr")]. I have asked the user what data is, but I haven't been able to figure it out myself and was therefore hoping for some help in this regard.

from bs4 import BeautifulSoup
import requests
from itertools import islice
import pandas as pd

r = requests.get(
    "http://www.federalreserve.gov/econresdata/researchdata/feds200628_1.html")
soup = BeautifulSoup(r.content)

h = data.find_all("th", scope="col") #Issue
final = [[t.th.text] + [ele.text for ele in t.find_all("td")] for t in h[-1].find_all_next("tr")]
headers = [th.text for th in h]

headers.insert(0,"dates")
df = pd.DataFrame(final,columns=headers)

print df

Upvotes: 0

Views: 44

Answers (1)

Vikas Ojha
Vikas Ojha

Reputation: 6950

from bs4 import BeautifulSoup
import requests
from itertools import islice

r = requests.get(
    "http://www.federalreserve.gov/econresdata/researchdata/feds200628_1.html")
soup = BeautifulSoup(r.content)

data = soup.find('table', attrs={'rules': 'all'})
h = data.find_all("th", scope="col") #Issue
final = [[t.th.text] + [ele.text for ele in t.find_all("td")] for t in h[-1].find_all_next("tr")]
headers = [th.text for th in h]

Upvotes: 1

Related Questions