CODEWITHSUNDEEP
pythonbeautifulsoup
Michael Queue
Michael Queue

Reputation: 1400

Scrape just the text, within an html element that has a class, using beautiful soup

I'm trying to scrape a page using BeatifulSoup

import urllib2
from bs4 import BeautifulSoup

url='http://www.xpn.org/playlists/xpn-playlist'
page = urllib2.urlopen(url)

soup = BeautifulSoup(page.read())

for link in soup.find_all("li", class_="song"):
    print link

The problem is the text I would like to return is not enclosed in it's own html tag

<li class="song"> <a href="/default.htm" onclick="return clickreturnvalue()
" onmouseout="delayhidemenu()" onmouseover="dropdownmenu(this, event, menu1, 
'100px','Death Vessel','Mandan Dink','Stay Close')">Buy</a>  
Chuck Ragan - Rotterdam - Folkadelphia Session</li>

What I want to return Chuck Ragan - Rotterdam - Folkadelphia Session

Bonus Points: The data returned is of the format Artist/Song/Album. What would be the proper data structure to use to store and manipulate this info?

Upvotes: 6

Views: 646

Answers (4)

Michael Queue
Michael Queue

Reputation: 1400

** Ended up using a named tuple for storage **

from bs4 import BeautifulSoup
import urllib2
from collections import namedtuple

url='http://www.xpn.org/playlists/xpn-playlist'
page = urllib2.urlopen(url)


soup = BeautifulSoup(page.read())

songs=[]
Song = namedtuple("Song", "artist name album")
for link in soup.find_all("li", class_="song"):
    song = Song._make(link.text.strip()[12:].split(" - "))
    songs.append(song)

for song in songs:
    print(song.artist, song.name, song.album)

Upvotes: 0

Aziz Alto
Aziz Alto

Reputation: 20331

Here is another way! (assuming li has 3 children. If not, change [2] to [1]):

>>> html = '''<li class="song"> <a href="/default.htm" onclick="return clickreturnvalue()
... " onmouseout="delayhidemenu()" onmouseover="dropdownmenu(this, event, menu1,
... '100px','Death Vessel','Mandan Dink','Stay Close')">Buy</a>
... Chuck Ragan - Rotterdam - Folkadelphia Session</li>'''

>>> from bs4 import BeautifulSoup as bs
>>> all_li = soup.findAll('li', class_='song')
>>> for li in all_li:
...     text = list(li.children)[2]
...     artist, song, album = text.split('-')
...     print artist, song, album
Chuck Ragan   Rotterdam   Folkadelphia Session

Upvotes: 1

Remi Guan
Remi Guan

Reputation: 22282

Try something like:

for link in soup.find_all("li", class_="song"):
    print link.text

Output:

Buy  Chuck Ragan - Rotterdam - Folkadelphia Session

Sure, if you want to remove Buy, you can use slice like this:

for link in soup.find_all("li", class_="song"):
    print link.text.strip()[5:]

The output is:

Chuck Ragan - Rotterdam - Folkadelphia Session

If you'd like save these string in a list:

[i.strip() for i in link.text.strip()[5:].split('-')]

Output:

['Chuck Ragan', 'Rotterdam', 'Folkadelphia Session']

For more info, you can check the document.

Upvotes: 1

JRodDynamite
JRodDynamite

Reputation: 12613

You could use something like this.

for l in soup.find_all("li", class_="song"):
    album = l.text.split("-")[2]
    song = l.text.split("-")[1]
    artist = l.text.split("-")[0].split(" ")[1]

Upvotes: 0

Related Questions