CODEWITHSUNDEEP
pythonseleniumxpath
GreenGodot
GreenGodot

Reputation: 6773

Get the nth occurence of an element using Xpath in Selenium while waiting

I am scraping a large static web page using Selenium. I know ahead of time how many <a> elements will be present on the page. Since it is a very large page, I want to be sure it's completely loaded before attempting to scrape it. My solution is to wait until the last <a> element has been loaded. I have tried using presence_of_element_located as shown here:

driver.get(url)
try:
    WebDriverWait(driver, 500).until(EC.presence_of_element_located(driver.find_elements_by_xpath('//*[@title="View recipe"]')[count]))
except TimeoutException:

But it throws an error:

Traceback (most recent call last):


File "/home/noname365/siteCrawler/test.py", line 28, in <module>
    WebDriverWait(driver, 500).until(EC.presence_of_element_located(driver.find_elements_by_xpath('//*[@title="View recipe on foodily.com"]')[10 -1]))
  File "/home/noname365/virtualenvs/env35/lib/python3.5/site-packages/selenium/webdriver/support/wait.py", line 71, in until
    value = method(self._driver)
  File "/home/noname365/virtualenvs/env35/lib/python3.5/site-packages/selenium/webdriver/support/expected_conditions.py", line 59, in __call__
    return _find_element(driver, self.locator)
  File "/home/noname365/virtualenvs/env35/lib/python3.5/site-packages/selenium/webdriver/support/expected_conditions.py", line 274, in _find_element
    return driver.find_element(*by)
TypeError: find_element() argument after * must be a sequence, not WebElement

What am I doing wrong here?

Upvotes: 1

Views: 521

Answers (2)

Kim Homann
Kim Homann

Reputation: 3249

Why not wait for the page to be loaded? Here's the C# code, should be similar in Python. The JavaScript part should be exactly the same.

protected void WaitForDocumentReadyStateComplete()
{
    try
    {
        new WebDriverWait(target.Driver, TimeSpan.FromSeconds(DefaultTimeoutInSeconds)).Until(
            d => ((IJavaScriptExecutor) d).ExecuteScript("return document.readyState").Equals("complete"));
        // Safari (Mac) sometimes hangs for 30 seconds then throws WebDriverTimeoutException => can safely be ignored
    }
    catch (Exception)
    {
        if (!target.IsSafari)
        {
            // MSIE (Win) sometimes throws "UnexpectedJavaScriptError" => Workaround: wait maximum time
            Thread.Sleep(DefaultTimeoutInSeconds * 1000);
        }
    }
}

Or you wait for the expected URL (might not work if you reload a page and the URL doesn't change):

new WebDriverWait(target.Driver, TimeSpan.FromSeconds(DefaultTimeoutInSeconds)).Until(
    ExpectedConditions.UrlMatches(MatchUrl));

Upvotes: 0

alecxe
alecxe

Reputation: 474071

presence_of_element_located() and other Expected Conditions, in the first and only argument, expect a tuple with a type of locator as the first item and the locator value as the second item:

from selenium.webdriver.common.by import By
EC.presence_of_element_located((By.XPATH, '//*[@title="View recipe"]'))

Upvotes: 2

Related Questions