Extract HTML tag data of content of its first tag matches

Question

I have just started learning Beautiful soup and I am extracting data from a webpage which has tags some like this:

<tr > < td class = "review2"
width = "23%" > Overall rating: < /td><td>  </td > < td class = "review3" > < img hight = "18px"
src = "/img/red_star.gif" width = "18px" / > < img hight = "18px" src = "/img/red_star.gif" width = "18px" / > < /td></tr >
<tr > < td class = "review2" > Effectiveness: < /td><td>  </td > < td class = "review3" > Highly Effective < /td></tr >

What I want is that if tag has text "Overall rating:" then I count the img['src']. The img['src'] gives me the rating of a particular object.

I really dont know how to proceed. I tried multiple ways.

tr = soup.find_all('tr')
td = []
for i in tr:
    td.append(i.find_all('td',class_='review3'))

Can anyone please help.

Thanks

Answer 1

If inside the <tr> aren't other images, you can use this example to get the count:

from bs4 import BeautifulSoup


html_doc = """
<tr> <td class = "review2"
width = "23%"> Overall rating: </td><td>  </td> <td class = "review3"> <img hight = "18px"
src = "/img/red_star.gif" width = "18px" /> <img hight = "18px" src = "/img/red_star.gif" width = "18px" /> </td></tr>
<tr> <td class = "review2"> Effectiveness: </td><td>  </td> <td class = "review3"> Highly Effective </td></tr>
"""

soup = BeautifulSoup(html_doc, "html.parser")

imgs = soup.select('td:contains("Overall rating:") ~ td > img')
print("Rating:", len(imgs))

Prints:

Rating: 2

---

EDIT: To scrape more info:

import requests
from bs4 import BeautifulSoup


url = "http://www.druglib.com/ratingsreviews/abilify/"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

for h2 in soup.select("h2"):
    print(h2.text)
    print(
        "Rating:",
        len(
            h2.find_next(class_="review3").find_all(
                lambda tag: tag.name == "img"
                and "red_star" in tag.get("src", "")
            )
        ),
    )
    print(
        "Review:",
        h2.find_next("td", text="Benefits:").find_next(class_="review3").text,
    )
    print(
        "Side effects:",
        h2.find_next(
            lambda tag: tag.name == "td" and "Reported Results" in tag.text
        )
        .find_next("td", text="Side effects:")
        .find_next(class_="review3")
        .text,
    )
    print("-" * 80)

Prints:

Abilify review by  26 year old female patient
Rating: 3
Review:  I didn't notice any benefit at all. Supposedly, it was to help keep my mood balanced.  More than anything, I was focused on the very apparent and prevalent side effects.
Side effects:  A very uncomfortable inner restlessness was the worst side effect.  I felt like I had to constantly get up and move around, but it didn't even help.  I couldn't sit still, had major problems sleeping at night, AND started gaining weight.
--------------------------------------------------------------------------------
Abilify review by  29 year old female patient
Rating: 8
Review:  I had severe depression with agitation and mixed-state bipolar hypomania. I was insomniac. I took abilify 5mg for 3 mos, and then I upped it to 10mg. 
Side effects:  I became drowsy, however, with adequate sleep (10 hrs) there were no hangover effects, unlike zyprexa, for instance. I also experienced some cognitive blunting- that is to say that I could not think very well on my feet
--------------------------------------------------------------------------------
Abilify review by  43 year old female patient
Rating: 10
Review:  Within 1 week of taking the cocktail of Abilify and Lexapro, a extremely unorganized person....became organized. I am now able to remember appointment times, keep up with my daily responsibilities...it has been a lifesaver
Side effects:  no side effects have been noticed
--------------------------------------------------------------------------------
Abilify review by  50 year old female patient
Rating: 2
Review:  While on abilify I can honestly say the depression resolved,
Side effects:   but it caused memory loss and again an incident involving retail theft. The retail theft occured while on just 5 mg of abilify. It effected my judgement/reasoning skills because I did not tell my doctor & allowed him to continue to increase dosage up to 15mg when I became expremely restless. Could not set down, paced for hours & called my doctor about this. Also during the 2 months I was taking this drug I charged a little over 15,000 on my credit cards that had no balance on them previously. If one did it was under $300. I still am currently unemployed & am having difficulty finding employement as a RN, due to my criminal record within the last year. If I had problems before taking antipsychotics, they were nothing compared to what I have now. 
--------------------------------------------------------------------------------
Abilify review by  50 year old male patient
Rating: 2
Review:  None due to the short time taking drug.
Side effects:  Headache first morning at 4AM that was relieved with Excedrin.  Migraine headache with vomiting that lasted 14 hours.  Couldn't go to work that day and stopped taking it.
--------------------------------------------------------------------------------
Abilify review by  52 year old female patient
Rating: 6
Review:  I've only been on it for a week but I've noticed a change already. I am more awake and it seems as though a fog has been lifted. I am thinking more clearly and don't feel so down.
Side effects:  None so far.
--------------------------------------------------------------------------------
Abilify review by  52 year old female patient
Rating: 6
Review:  I've only been on it for a week but I've noticed a change already. I am more awake and it seems as though a fog has been lifted. I am thinking more clearly and don't feel so down.
Side effects:  None so far.
--------------------------------------------------------------------------------
Abilify review by  52 year old female patient
Rating: 6
Review:  I've only been on it for a week but I've noticed a change already. I am more awake and it seems as though a fog has been lifted. I am thinking more clearly and don't feel so down.
Side effects:  None so far.
--------------------------------------------------------------------------------