CODEWITHSUNDEEP
pythoncssseleniumweb-scraping
ohbrobig
ohbrobig

Reputation: 966

Selenium Extraction Problems: Waits/Not Finding Elements

In both chrome and firefox, everything is fine up until I need to extract text. I get this error:

h3 = next(element for element in h3s if element.is_displayed())
StopIteration

I even added a fluent wait.

browser = webdriver.Firefox()
browser.get('https://www.voilanorbert.com/')
inputElement = browser.find_element_by_id("form-search-name")
inputElement.send_keys(leadslist[i][0])
inputElement = browser.find_element_by_id("form-search-domain")
inputElement.send_keys(leadslist[i][1])
searchbutton = browser.find_element_by_name("search")
searchbutton.click()

wait = WebDriverWait(browser, 20)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.results")))
wait2 = WebDriverWait(browser, 3000, poll_frequency=100, ignored_exceptions=[ElementNotVisibleException])
wait2.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "h3.one")))
h3s = browser.find_elements_by_css_selector('h3.one')
h3 = next(element for element in h3s if element.is_displayed())
result = h3.text

I think its because its not actually extracting anything, so its just an empty list. Some pictures that will probably help: This is the before picture: enter image description here

This is the after picture: enter image description here

I need to extract what is in the "text-center displayed" class of the "result" class.

enter image description here

Upvotes: 0

Views: 91

Answers (1)

FDM
FDM

Reputation: 658

The answer is fairly simple, you just need a different selector when waiting for the search result. The approach below (C#) works perfectly, it'll reduce your code with a few lines.

One "result DIV" becomes visible when the search is done. It is the only element with the "text-center displayed" class, so that's all your selector needs. Once such a DIV is displayed, you know where to pinpoint the H3 element (it's a child of said DIV). So simply wait for the below element to become visible after you've clicked the search button:

        IWebElement headerResult = w.Until(ExpectedConditions.ElementIsVisible(By.CssSelector("div[class=\"text-center displayed\"] h3")));
        string result = headerResult.Text;

Upvotes: 1

Related Questions