CODEWITHSUNDEEP
pythonbeautifulsoup
Jolijt Tamanaha
Jolijt Tamanaha

Reputation: 333

using BeautifulSoup to find divs within a div

I'm trying to have BeautifulSoup look for all five divs with the class "blog-box" and then look within each one of those divs and find the div with the class "date" and the class "right-box" and then print those. I need it to print the date and then print the related text immediately so that's why I can't just look for the "date" and "right-box" divs directly. 

for i in xrange(3, 1, -1):
       page = urllib2.urlopen("http://web.archive.org/web/20090204221349/http://www.americansforprosperity.org/nationalblog?page={}".format(i))
       soup = BeautifulSoup(page.read())
       snippet = soup.find_all('div', attrs={'class': 'blog-box'})
       print snippet
       for div in snippet:
           date =  soup.find('div', attrs={'class': 'date'})
           text = soup.find('div', attrs={'class': 'right-box'})
           print date.text
           print text.text

But I run this and it prints the first date and text divs five times and then stops.

Upvotes: 3

Views: 2201

Answers (1)

shx2
shx2

Reputation: 64298

It seems that you accidentally use soup inside the inner loop, instead of the loop variable div. Try:

for ...:
   ...
   for div in snippet:
       date = div.find('div', attrs={'class': 'date'})  # <-- changed here
       text = div.find('div', attrs={'class': 'right-box'})  # <--changed here
       print date.text
       print text.text

Upvotes: 2

Related Questions