CODEWITHSUNDEEP
pythonhtmlbeautifulsouphtml-parsing
SpicyClubSauce
SpicyClubSauce

Reputation: 4256

Pulling specific (text) spaced between HTML tag during BeautifulSoup

I'm trying to pull something that is categorized as (text) when I look at it in "Inspect Element" mode:

<div class="sammy"
  <div class = "sammyListing">
    <a href="/Chicago_Magazine/blahblahblah">
       <b>BLT</b>
       <br>
       " 
           Old Oak Tap" <---**THIS IS THE TEXT I WANT**
       <br>
       <em>Read more</em>
    </a>
  </div>
</div>

This is my code thus far, with the line in question being the bottom list comprehension at the end:

STEM_URL = 'http://www.chicagomag.com'
BASE_URL = 'http://www.chicagomag.com/Chicago-Magazine/November-2012/Best-Sandwiches-Chicago/'

soup = BeautifulSoup(urlopen(BASE_URL).read())
sammies = soup.find_all("div", "sammy")
sammy_urls = []
for div in sammies:
    if div.a["href"].startswith("http"):
        sammy_urls.append(div.a["href"])
    else:
        sammy_urls.append(STEM_URL + div.a["href"])
    restaurant_names = [x for x in div.a.content]

I've tried div.a.br.content, div.br, but can't seem to get it right.

If suggesting a RegEx way, I'd also really appreciate a nonRegEx way if possible.

Upvotes: 1

Views: 59

Answers (1)

alecxe
alecxe

Reputation: 473863

Locate the b element for every listing using a CSS selector and find the next text sibling:

for b in soup.select("div.sammy > div.sammyListing > a > b"):
    print b.find_next_sibling(text=True).strip()

Demo:

In [1]: from urllib2 import urlopen

In [2]: from bs4 import BeautifulSoup

In [3]: soup = BeautifulSoup(urlopen('http://www.chicagomag.com/Chicago-Magazine/November-2012/Best-Sandwiches-Chicago/'))

In [4]: for b in soup.select("div.sammy > div.sammyListing > a > b"):
   ...:     print b.find_next_sibling(text=True).strip()
   ...:     
Old Oak Tap
Au Cheval
...
The Goddess and Grocer
Zenwich
Toni Patisserie
Phoebe’s Bakery

Upvotes: 1

Related Questions