CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scraping
Hooman Bahreini
Hooman Bahreini

Reputation: 15559

How to use nested selenium selectors

I am using selenium + chrome web driver to load a dynamic page:

self.driver.get(url)
time.sleep(3)  # not sure if I need to add this wait, so the .js loads the page?

Once the page is loaded, I want to get a list of all cards available on the page and then iterate through each card and get its title:

cards = self.driver.find_elements_by_css_selector('div.my-card')
for card in cards:
    title = card.find_element_by_css_selector('h2.title::text').get() # <-- does not work
    desc = card.find_element_by_css_selector('div.desc::text').get() # <-- does not work
    # more fields that I need within this card

find_elements_by_css_selector seems to be a driver method... I am not sure how to apply these selector to card (the type card is WebElement).

Sample page:

<div class='my-card'>
    <h2 class='title'>title 1</h2>
    <div class='desc'>desc 1</div>
</div>
<div class='my-card'>
    <h2 class='title'>title 2</h2>
    <div class='desc'>desc 2</div>
</div>
<div class='my-card'>
    <h2 class='title'>title 3</h2>
    <div class='desc'>desc 3</div>
</div>

Upvotes: 1

Views: 2437

Answers (2)

sytech
sytech

Reputation: 40861

What you have works as-is. Using your given example HTML:

In [19]: for card in cards:
    ...:     title_elem = card.find_element_by_css_selector('h2.title')
    ...:     print(title_elem.text)
    ...:
title 1
title 2
title 3

In [20]: card
Out[20]: <selenium.webdriver.remote.webelement.WebElement (session="534c4be3a233a0aa963f541550ac7861", element="b56dcaf3-9b38-4e4c-aae8-99005ac9840b")>

So. Your expectation of using nested selenium selectors is correct. Some other assumption must be throwing you off.

Upvotes: 1

idontknow
idontknow

Reputation: 438

You can combine the two into one query (CSS) selector like so by using the combinator child or descendent selector.

If the h2 element is a child and a descendent of the card element:

self.driver.find_elements_by_css_selector('div.my-card > h2.title')

If the h2 element is only a descendent of the card element:

self.driver.find_elements_by_css_selector('div.my-card h2.title')

Find out more about CSS combinators here.

Upvotes: 3

Related Questions