CODEWITHSUNDEEP
pythonbeautifulsoup
LBR
LBR

Reputation: 113

Trouble parsing HTML using BeautifulSoup

I'm trying to use BeautifulSoup to parse some HTML in Python. Specifically, I'm trying to create two arrays of soup objects: one for the dates of postings on a website, and one for the postings themselves. However, when I use findAll on the div class that matches the postings, only the initial tag is returned, not the text inside the tag. On the other hand, my code works just fine for the dates. What is going on??

# store all texts of posts
texts = soup.findAll("div", {"class":"quote"})

# store all dates of posts
dates = soup.findAll("div", {"class":"datetab"})

The first line above returns only

<div class="quote">

which is not what I want. The second line returns

<div class="datetab">Feb<span>2</span></div>

which IS what I want (pre-refining).

I have no idea what I'm doing wrong. Here is the website I'm trying to parse. This is for homework, and I'm really really desperate.

Upvotes: 0

Views: 613

Answers (3)

Ori
Ori

Reputation: 4222

Which version of BeautifulSoup are you using? Version 3.1.0 performs significantly worse with real-world HTML (read: invalid HTML) than 3.0.8. This code works with 3.0.8:

import urllib2
from BeautifulSoup import BeautifulSoup

page = urllib2.urlopen("http://harvardfml.com/")
soup = BeautifulSoup(page)
for incident in soup.findAll('span', { "class" : "quote" }):
    print incident.contents

Upvotes: 3

karlcow
karlcow

Reputation: 6972

There's a python port of Tumblr that you can use to read documents.

from tumblr import Api

api = Api('harvardfml.com')
freq = {}
posts = api.read()
for post in posts:
   #do something here

for your bogus findAll, without the actual source code of your program it is hard to see what is wrong.

Upvotes: 0

icktoofay
icktoofay

Reputation: 129129

That site is powered by Tumblr. Tumblr has an API.

Upvotes: 1

Related Questions