CODEWITHSUNDEEP
pythonseleniumxpathhref
Sashaank
Sashaank

Reputation: 964

how to extract links from multiple divs with the same class name with xpath

I am working on a selenium project. I have a web page with multiple divs with the same class name such as this

<div class="usr_blk hid">...</div>
<div class="usr_blk hid">...</div>
<div class="usr_blk hid">...</div>
<div class="usr_blk hid">...</div>
<div class="usr_blk hid">...</div>
<div class="usr_blk hid">...</div>
<div class="usr_blk hid">...</div>
<div class="usr_blk hid">...</div>
<div class="usr_blk hid">...</div>

inside each div is the following content

<div class="usr_blk hid">
    <div class="image_info">
        <table>
            <tbody>
                <tr>
                    <td>
                        <a href="the link I want to scrape">Link</a>
                    </td>
                </tr>
           </tbody>
        </table>
    </div>
</div>

I want to extract the link from the a tag inside each div. This is what I tried so far

list_xpath = "//div[@class='usr_blk hid']//div[@class='image_info']//a"
list_raw = driver.find_elements_by_xpath(list_xpath)
lst = [link.get_attribute('href') for link in list_raw]

But this results in an empty list. I am not sure what I am doing wrong.

Thanks in advance

I cannot share the website link with you because of company policy.

Upvotes: 1

Views: 612

Answers (1)

frianH
frianH

Reputation: 7563

Try using WebDriverWait.

.visibility_of_all_elements_located return a list of web element:

wait = WebDriverWait(driver, 20)
list_raw = wait.until(EC.visibility_of_all_elements_located((By.XPATH, 'your_xpath')))

Import:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Upvotes: 1

Related Questions