CODEWITHSUNDEEP
pythonseleniumselenium-webdriver
Efan Mutembo
Efan Mutembo

Reputation: 15

How can I turn a webdriver element into text consistently

I'm scraping a table from @ https://www.premierleague.com/tables?co=1&se=363&ha=-1 and am getting this error when I turn the web driver object that corresponds to td into text. The error is only there for the first few times I run this code then it disappears!!

Here is my code:


#Click to open the season dropdown

driver.find_elements(By.CLASS_NAME,'current')[1].click()
WebDriverWait(driver,5)# wait for dropdown to apear

#Click to chose a season
season_tags[2].click()


"""
2. Scrape the data from the now open season x table

"""

WebDriverWait(driver,20)# wait for table to appear

cells = driver.find_elements(By.TAG_NAME,'td')#find all table cells
WebDriverWait(driver,20)
cells_text = [i.text for i in cells]#turn into text

table = [cells_text[i+2:i+11] for i in range(0,len(cells_text))][::13]

And I get this error:

     21 cells = driver.find_elements(By.TAG_NAME,'td')#find all table cells
     22 WebDriverWait(driver,20)
---> 23 cells_text = [i.text for i in cells]#turn into text
     25 table = [cells_text[i+2:i+11] for i in range(0,len(cells_text))][::13]

File ~\Anaconda\lib\site-packages\selenium\webdriver\remote\webelement.py:84, in WebElement.text(self)
     81 @property
     82 def text(self) -> str:
     83     """The text of the element."""
---> 84     return self._execute(Command.GET_ELEMENT_TEXT)['value']

File ~\Anaconda\lib\site-packages\selenium\webdriver\remote\webelement.py:396, in WebElement._execute(self, command, params)
    394     params = {}
    395 params['id'] = self._id
--> 396 return self._parent.execute(command, params)

File ~\Anaconda\lib\site-packages\selenium\webdriver\remote\webdriver.py:429, in WebDriver.execute(self, driver_command, params)
    427 response = self.command_executor.execute(driver_command, params)
    428 if response:
--> 429     self.error_handler.check_response(response)
    430     response['value'] = self._unwrap_value(
    431         response.get('value', None))
    432     return response

File ~\Anaconda\lib\site-packages\selenium\webdriver\remote\errorhandler.py:243, in ErrorHandler.check_response(self, response)
    241         alert_text = value['alert'].get('text')
    242     raise exception_class(message, screen, stacktrace, alert_text)  # type: ignore[call-arg]  # mypy is not smart enough here
--> 243 raise exception_class(message, screen, stacktrace)

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document```

Upvotes: 0

Views: 67

Answers (1)

Prophet
Prophet

Reputation: 33361

First of all this WebDriverWait(driver,20) is not a pause command and this command will not cause to any delay / pause.
wait = WebDriverWait(driver, 20) this creates a wait object that can be used to wait for several expected conditions.
Now, the StaleElementReferenceException means that the element was changed.
This can be caused by 1 of the following 2:

  1. The page is continue changing after you performed this line cells = driver.find_elements(By.TAG_NAME,'td'). If so you can add some pause like time.sleep(5) before this line cells = driver.find_elements(By.TAG_NAME,'td').
  2. cells may contain rows that are initially not rendered, so that by accessing them the page is actually scrolled. This causes these elements to be changed. If so you need to get the cells elements again during the iteration thro the for loop.

Upvotes: 1

Related Questions