CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Amar
Amar

Reputation: 57

Beautiful Soup error: 'NoneType' object has no attribute 'find_all'

I am getting this error:

---> 14 content = s.find_all('p')
AttributeError: 'NoneType' object has no attribute 'find_all'

While running the script below:

r = requests.get('https://www.marketsandmarkets.com/Market-Reports/rocket-missile-market-203298804.html/')
soup = BeautifulSoup(r.content, 'html.parser')
s = soup.find('div', class_='entry-content') 
content = s.find_all('p')
print(content)

It's working for some URLs, but for other URLs it gives an attribute error. Not sure why this is happening.

Upvotes: 1

Views: 2962

Answers (1)

joanis
joanis

Reputation: 12280

When soup.find does not find anything, it returns None. Guard your s.find_all call with if s: or if s in not None:.

So your code becomes:

r = requests.get('https://www.marketsandmarkets.com/Market-Reports/rocket-missile-market-203298804.html/')
soup = BeautifulSoup(r.content, 'html.parser')
s = soup.find('div', class_='entry-content') 
if s is not None:
    content = s.find_all('p')
    print(content) 
else:
    print("Didn't find what I was looking for...")

or maybe:

content = s.find_all('p') if s is not None else []
print(content)

Ref: https://crummy.com/software/BeautifulSoup/bs4/doc/#find

Upvotes: 2

Related Questions