CODEWITHSUNDEEP
pythonbeautifulsoup
bill999
bill999

Reputation: 2529

Python - BeautifulSoup - how to check if ResultSet contains an element

I am doing some web-scraping, but I've come up with something that I can't figure out. Basically, I need to check if the 0'th element of my ResultSet element releaseDate contains 'content' as in 

[<meta content="1992-09-11" itemprop="datePublished"/>]

But when 'content' is not in the tag, I get an error like

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "imdbQuestion.py", line 18, in <module>
if releaseDate[0]['content']:
File "build/bdist.macosx-10.8-intel/egg/bs4/element.py", line 879, in __getitem__
KeyError: 'content'

How can I check if 'content' is in releaseDate without causing an error?

Additionally, how can I extract whatever I want out of ResultSet objects?

The full code is:

import requests
from bs4 import BeautifulSoup

file = codecs.open('imdb.txt', 'w', encoding = 'utf-8')

#iterate through last value
for increment in range(7,10):
    imdbNum = '015008' + str(increment)
    url = 'http://www.imdb.com/title/tt' + imdbNum

    urlCode = requests.get(url)
    soup = BeautifulSoup(urlCode.content)

    #get release date
    releaseDate = soup.findAll(attrs={'itemprop':'datePublished'})
    abc = releaseDate
        #error checking - assign '.' to releaseDate if releaseDate[0] is blank
        #if not blank, check if 'content' is in releaseDate[0]. if so, we are good.  if not, assign 'CHECK' to releaseDate[0]
    if releaseDate:
        if releaseDate[0]['content']:
            releaseDate = releaseDate[0]['content']
        else:
            releaseDate = 'CHECK'
    else:
        releaseDate = '.'

    print releaseDate
    file.close()

Upvotes: 2

Views: 12111

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1122382

Test against the Tag.attrs dictionary:

if releaseDate:
    if 'content' in releaseDate[0].attrs:
        releaseDate = releaseDate[0]['content']
    else:
        releaseDate = 'CHECK'

or use the dict.get() method on that attribute:

if releaseDate:
    releaseDate = releaseDate[0].attrs.get('content', 'CHECK')

Quick demo:

>>> import requests
>>> from bs4 import BeautifulSoup
>>> imdbNum = '0150087'
>>> url = 'http://www.imdb.com/title/tt' + imdbNum
>>> urlCode = requests.get(url)
>>> soup = BeautifulSoup(urlCode.content)
>>> releaseDate = soup.findAll(attrs={'itemprop':'datePublished'})
>>> releaseDate[0]
<meta content="1966-04" itemprop="datePublished"/>
>>> releaseDate[0].attrs
{'content': '1966-04', 'itemprop': 'datePublished'}
>>> 'content' in releaseDate[0].attrs
True
>>> releaseDate[0].attrs.get('content', 'CHECK')
'1966-04'

Upvotes: 3

Related Questions