CODEWITHSUNDEEP
pythonselenium
A.Hamza
A.Hamza

Reputation: 229

selenium contains text not able to find the element in a dialog box

I needed to scrape a site that contains a list. Clicking any list element shows a modal dialog which contains some text that I want to scrape. Doing this in a loop gets stale element exception so this is what I did initially

elems = browser.find_elements_by_xpath("//a[@data-toggle='modal']")
temp_names = []
for elem in elems:
    temp_names.append(elem.text)

where clicking on this anchor tag element shows the modal dialog so I stored the text of all these elements in a list and finally I did this:

for temp_name in temp_names:
    print(temp_name)
    modals= browser.find_element_by_xpath("//a[contains(text(),'"+temp_name+"')]")
    modals.click()
    time.sleep(1)
    # rest of the scraping code

Now I get No element exception error even though temp_name prints right text. The structure of the webpage is as follows:

<div class="...">
    <h4 class="...">
        <a href=# data-toggle="modal" data-target="#(target modal dialogue id)">Text</a>
    </h4>
    .   .   .
    (Some Other tags)
    .   .   .
</div>
<div id =(modal dialogue id) class="..." role="dialog">
    .   .   .
    (Some text I want to Scrape)
    .   .   .    
</div>

Text in anchor tag is the one I am looking for. I plan to find the element by text and then click it and then go back to the original url using browser.get(URL) at end of loop and find again the second element and so on. I don't understand why it is not able to find the element since I just got the element text in the previous loop. Also if there is a better way to do it please share.

Note: I can't scrape directly from modal dialog div unless I click first otherwise the text returned would be empty.

Edit:

Following is my current code:

chromedriver = 'C:\\chromedriver.exe'
chop = webdriver.ChromeOptions()
chop.add_extension('C:\\AdBlock_v3.38.1.crx')
time.sleep(5)
browser = webdriver.Chrome(chromedriver, chrome_options = chop)


for i in range(1,22):
    browser.get("http://pasha.org.pk/members/page/"+str(i)+"/")
    time.sleep(1)
    elems = browser.find_elements_by_xpath("//a[@data-toggle='modal']")
    print(len(elems))
    temp_names = []
    for elem in elems:
        temp_names.append(elem.text)
    current = browser.current_url
    for temp_name in temp_names:
        print()
        print(temp_name)
        print()
        modals= browser.find_element_by_xpath(f"//a[contains(text(), '{temp_name}')]")
        modals.click()
        time.sleep(1)
        # elem2 = browser.find_element_by_xpath("//button[@class='close']")
        # time.sleep(1)
        browser.get(current)

Following is the snippet of the error I get:

Error when code runs

Upvotes: 0

Views: 648

Answers (1)

C. Peck
C. Peck

Reputation: 3717

Couple things you could try--I think your quotes might be messing up the syntax a bit, but if that is the problem I'm not sure why it would work for the first loop....

modals= browser.find_element_by_xpath(f"//a[contains(text(), '{temp_name}')]")

modals= browser.find_element_by_xpath("//a[contains(text(), '" + temp_name + "')]")

Do either of those definitions work better?

If not, can you upload the full error text you are receiving?

Edit to address OP's clarification: "I need to click all tags with data-toggle='modal'"

For me, the following xPath returns 20 <a> elements with data-toggle='modal'

modals= browser.find_element_by_xpath("//a[@data-toggle='modal']")

Upvotes: 1

Related Questions