CODEWITHSUNDEEP
pythonseleniumweb-scraping
HrkBrkkl
HrkBrkkl

Reputation: 673

How to wait for presence of element and particular text in element?

I need to wait for the presence of an element with specific text in it. I want to fetch the info from that element in the moment the element is present and has text in it. I appears sometime after submitting a form and normaly fills with info a little later. My current solution looks like this:

wait = WebDriverWait(self.driver, MAXIMUM_LOAD_TIME)
    try:
        wait.until(ec.presence_of_element_located((By.ID,"IdOfElement")))
        wait.until(ec.text_to_be_present_in_element((By.ID, "IdOfElement"), "theText"))
        data = self._extract_data()
    except TimeoutException:
        raise WebsiteTimeoutError(MAXIMUM_LOAD_TIME)

This runs perfectly in >99% of the cases but now it happened that i got an error on ec.text_to_be_present_in_element.

The error was:

  File "my/Path", line 436, in  _scrape_quote
    wait.until(ec.text_to_be_present_in_element((By.ID, "IdOfElement"), "theText"))
  File "C:\Program Files (x86)\Python38-32\lib\site-packages\selenium\webdriver\support\wait.py", line 71, in until
    value = method(self._driver)
  File "C:\Program Files (x86)\Python38-32\lib\site-packages\selenium\webdriver\support\expected_conditions.py", line 210, in __call__
    return self.text in element_text
TypeError: argument of type 'NoneType' is not iterable

Apparently the element dissapeared again. Is my assumption right? What is the best way to fix this?

Upvotes: 0

Views: 813

Answers (1)

Nic Laforge
Nic Laforge

Reputation: 1876

If you take a look at the text_to_be_present_in_element implementation it assume the value is never None.

    def __call__(self, driver):
        try:
            element_text = _find_element(driver, self.locator).text
            return self.text in element_text
        except StaleElementReferenceException:
            return False

If the value of element_text is None (which can be sometime), it will throw you an exception

TypeError: argument of type 'NoneType' is not iterable

Now your code is a little cumbersome as you need to ensure the element is present first, then retrieve the same element to find the proper value. There is no class in the EC module that offer both at the same time. So how about implementing your class.

You can implement a class that will ensure it is present and at the same time fix the issue to handle the case of None in the text return.

class FindPresenceOfTextInElement(object):
    def __init__(self, locator, text):
        self.locator = locator
        self.text = text

    def __call__(self, driver):
        try:
            text = driver.find_element(*self.locator).text
            return self.text in text
        except (NoSuchElementException, StaleElementReferenceException, TypeError):
            return False

Then your code becomes:

wait = WebDriverWait(self.driver, MAXIMUM_LOAD_TIME)
    try:
        wait.until(FindPresenceOfTextInElement((By.ID,"IdOfElement")))
        data = self._extract_data()
    except TimeoutException:
        raise WebsiteTimeoutError(MAXIMUM_LOAD_TIME)

You will need the following imports:

from selenium.common.exceptions import NoSuchElementException, StaleElementReferenceException

UPDATE

AS the above is the best approach, I believe, you could have work around it by adding the exception handler in the wait obj as follow:

wait = WebDriverWait(driver, 30, ignored_exceptions=(TypeError, ))

If you add the NoSuchElementException as well, you basically added to ensure element is present too.

Upvotes: 1

Related Questions