CODEWITHSUNDEEP
python-3.xweb-scrapingattributeerror
wus
wus

Reputation: 13

Webscraping with python to extract data

I am using following code. Everything works except the 'affiliation' part. It returns an error: AttributeError: 'NoneType' object has no attribute 'text' Without the .text, it returns everything--whole code inside the class

import requests
import bs4
import re

headers = {'User-Agent':'Mozilla/5.0'}

url = 'http://pubs.acs.org/toc/jacsat/139/5'
html = requests.get(url, headers=headers)

soup = bs4.BeautifulSoup(html.text, 'lxml')

tags = soup.findAll('a', href=re.compile("full"))

for tag in tags:
    new_url = tag.get('href', None)
    newurl = 'http://pubs.acs.org' + new_url
    newhtml = requests.get(newurl, headers=headers) 
    newsoup = bs4.BeautifulSoup(newhtml.text, 'lxml')

    article_title = newsoup.find(class_="articleTitle").text
    print(article_title)

    affiliations = newsoup.find(class_="affiliations").text
    print(affiliations)

    authors = newsoup.find(id="authors").text
    print(authors)

    citation_year = newsoup.find(class_="citation_year").text
    print(citation_year)

    citation_volume = newsoup.find(class_="citation_volume").text
    print(citation_volume)

    citation = newsoup.find(id="citation").text
    print(citation)

    pubdate = newsoup.find(id="pubDate").text
    print(pubdate)

Upvotes: 1

Views: 145

Answers (1)

Philippe Oger
Philippe Oger

Reputation: 1098

This exception was triggered because it did not find any element with the class "affiliation". I have checked and could not find any element with this class value in the source HTML (or any other attribute to that matter) in the first url your script scrapes.

I would catch the error to avoid your script to break and return None or a default string when it does not find the element.

Something like that would work:

try:
    affiliations = newsoup.find(class_="affiliations").text
    print(affiliations)
except AttributeError:
    affiliations = None

Upvotes: 1

Related Questions