CODEWITHSUNDEEP
htmlpython-3.xseleniumweb-scrapingbeautifulsoup
DanLee
DanLee

Reputation: 349

Python 3: How to web scrape text from div that contains multiple class values

I'm trying to web scrape a website (Here is the link to website), but the div in the page seems to have multiple class attributes which is making me hard to scrape the data. I tried to look for historical questions posted on Stackoverflow, but could not find an answer that I wanted. The below is part of the code I extracted from the website:

<div data-reactid="118">
  <div class="ue-ga base_ ue-jk" style="margin-left:-24px;margin-bottom:;" data-reactid="119">
    <div style="display: flex; flex-direction: column; width: 100%; padding-left: 24px;" data-reactid="120">
      <div class="ue-a3 ue-ap ue-a6 ue-gb ue-ah ue-n ue-f5 ue-ec ue-gc ue-gd ue-ge ue-gf base_ ue-jv ue-gz ue-h0 ue-h1" data-reactid="121">
        <div class="ue-a6 ue-bz ue-gb ue-ah ue-gg ue-gh ue-gi" data-reactid="122">
          <div class="ue-bn ue-bo ue-cc ue-bq ue-g9 ue-bs" title="Want to extract this part" data-reactid="123">
            Want to extract this part
          </div>
        </div>
      </div>
    </div>
  </div>
</div>

What I want to extract is the text where it states "Want to extract this part". I did think of scraping the data through data-reactid, but different pages have different data-reactid number assigned so wasn't a good idea. I also want to inform that class names are not unique.

Can anyone guide me through this? Much appreciated.

Upvotes: 4

Views: 4868

Answers (5)

Sers
Sers

Reputation: 12255

Menus:

enter image description here
- all menus to use in loop, css selector: div.base_ h3
- menu by name, xpath: //div[contains(@class,'base_')]//h3[.='Big Mac® Bundles']

Food Cards

enter image description here
- titles, css selector: div[title]
- titles, xpath: //div[./div[@title]]/div[@title]
- prices, xpath: //div[./div[@title]]//span
If you want to loop:

cards = driver.find_elements_by_xpath("//div[./div[@title]]")
for card in cards:
     title = card.find_element_by_css_selector("div[title]")
     price = card.find_element_by_css_selector("span")
     #or using xpath
     #title = card.find_element_by_xpath("./div[@title]")
     #price = card.find_element_by_xpath(".//span")

Category menu:

enter image description here
- all categories, css selector: a[href*='category']

Upvotes: 1

undetected Selenium
undetected Selenium

Reputation: 193108

As per the HTML you have shared to extract the text Want to extract this part as the element is a React element you have to induce WebDriverWait for the element to be visible and you can use either of the following solution:

  • Using title attribute:

    myText = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.base_ div[title]"))).get_attribute("title")

  • Using innerHTML:

    myText = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.base_ div[title]"))).get_attribute("innerHTML")

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Upvotes: 0

mala
mala

Reputation: 11

you can use jQuery as below.

$("div[title=Want to extract this part]").text();

Upvotes: 1

Sohan Das
Sohan Das

Reputation: 1620

This may help you

from bs4 import BeautifulSoup
html = """<div data-reactid="118">
<div class="ue-ga base_ ue-jk" style="margin-left:-24px;margin-bottom:;" data-reactid="119">
<div style="display: flex; flex-direction: column; width: 100%; padding-left: 24px;" data-reactid="120">
  <div class="ue-a3 ue-ap ue-a6 ue-gb ue-ah ue-n ue-f5 ue-ec ue-gc ue-gd ue-ge ue-gf base_ ue-jv ue-gz ue-h0 ue-h1" data-reactid="121">
    <div class="ue-a6 ue-bz ue-gb ue-ah ue-gg ue-gh ue-gi" data-reactid="122">
      <div class="ue-bn ue-bo ue-cc ue-bq ue-g9 ue-bs" title="Want to extract this part" data-reactid="123">
        Want to extract this part
      </div>
    </div>
  </div>
</div>
</div>
</div>"""

soup = BeautifulSoup(html,'html.parser')
tag = soup.find('div', attrs={'class':'ue-bn'})
text = (''.join(tag.stripped_strings))
print (text)

Upvotes: 1

itodd
itodd

Reputation: 2373

If the classes always remain the same for that specific element on each page you can target it with this selector:

.ue-bn.ue-bo.ue-cc.ue-bq.ue-g9.ue-bs

However, there are many other selectors you could use but it all depends on if they are unique and consistent across pages.

Upvotes: 1

Related Questions