CODEWITHSUNDEEP
pythonbeautifulsoup
SATXbassplayer
SATXbassplayer

Reputation: 111

BeautifulSoup: how to scrape meta tag description content

i am trying to scrape the contents of a website's meta description.

example: 

<meta name="description" content="This is the home page meta description.">

the output that i'm looking for is: "This is the home page meta description."

my code is:

raw_html = simple_get(companyUrl)
html = BeautifulSoup(raw_html, 'html.parser')
x = html.select('meta', {'name' : 'description'})  ## this line errors out

can someone point me in the right direction?

(also - is it my imagination, or are BeautifulSoup tutorials/documentation not up to the level of other languages/applications?)

Upvotes: 5

Views: 2696

Answers (2)

Artyom Vancyan
Artyom Vancyan

Reputation: 5390

Using BeautifulSoup

from bs4 import BeautifulSoup

html = """<meta name="description" content="This is the home page meta description.">"""

soup = BeautifulSoup(html, 'html.parser')
content = soup.find('meta', {'name':'description'}).get('content')
print(content)  # STDOUT: This is the home page meta description.

An alternate way using regex

import re

content = re.findall(r"content=\"(.*?)\"", html)

NOTE: regex parses faster and findall will return list of values of all content attributes defined at given html

Upvotes: 2

Martin Gergov
Martin Gergov

Reputation: 1658

You have to use a css selector like so:

x = html.select('meta[name="description"]')
print(x[0].attrs["content"])

Read more about css selectors here:

Upvotes: 4

Related Questions