CODEWITHSUNDEEP
pythonseleniumscreen-scraping
knguyen
knguyen

Reputation: 3074

Selenium clicks a link just once, calling click() another time returns error

I'm new to Selenium and just managed to write these codes. I want to scrap the data on the tables by clicking the '>' link at the bottom right. The first click works but the next two don't. What am I missing? Thanks.

# coding: utf-8
from selenium import webdriver
from selenium.webdriver.common.by import By

browser = webdriver.Firefox()
browser.get('http://s.cafef.vn/Lich-su-giao-dich-HSG-1.chn')

next_page_link = browser.find_element_by_partial_link_text('>')
next_page_link.click()
next_page_link = browser.find_element_by_partial_link_text('>')
next_page_link.click()
next_page_link = browser.find_element_by_partial_link_text('>')
next_page_link.click()

Here is the exception

Traceback (most recent call last):
  File "cafef.py", line 13, in <module>
    next_page_link.click()
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.40.0-py2.7.egg/selenium/webdriver/remote/webelement.py", line 59, in click
    self._execute(Command.CLICK_ELEMENT)
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.40.0-py2.7.egg/selenium/webdriver/remote/webelement.py", line 369, in _execute
    return self._parent.execute(command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.40.0-py2.7.egg/selenium/webdriver/remote/webdriver.py", line 164, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium-2.40.0-py2.7.egg/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: u'Element is no longer attached to the DOM' ; Stacktrace: 
    at fxdriver.cache.getElementAt (resource://fxdriver/modules/web_element_cache.js:7613)
    at Utils.getElementAt (file:///tmp/tmpIDOSdW/extensions/[email protected]/components/command_processor.js:7210)
    at fxdriver.preconditions.visible (file:///tmp/tmpIDOSdW/extensions/[email protected]/components/command_processor.js:8223)
    at DelayedCommand.prototype.checkPreconditions_ (file:///tmp/tmpIDOSdW/extensions/[email protected]/components/command_processor.js:10861)
    at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmpIDOSdW/extensions/[email protected]/components/command_processor.js:10878)
    at DelayedCommand.prototype.executeInternal_ (file:///tmp/tmpIDOSdW/extensions/[email protected]/components/command_processor.js:10883)
    at DelayedCommand.prototype.execute/< (file:///tmp/tmpIDOSdW/extensions/[email protected]/components/command_processor.js:10825)

Upvotes: 3

Views: 2470

Answers (2)

unutbu
unutbu

Reputation: 879471

The second next_page_link.click() call is occurring before the browser has loaded the next page. Add a wait.until with EC.element_to_be_clickable:

from selenium import webdriver
import selenium.webdriver.support.ui as UI
from selenium.webdriver.common.by import By
import selenium.webdriver.support.expected_conditions as EC 
import contextlib

with contextlib.closing(webdriver.Firefox()) as browser:
    browser.get('http://s.cafef.vn/Lich-su-giao-dich-HSG-1.chn')
    wait = UI.WebDriverWait(browser, 10)
    for i in range(3):            
        next_page_link = wait.until(
            EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, '>')))
        next_page_link.click()

Upvotes: 4

mizl&#39;
mizl&#39;

Reputation: 41

Usually the anwser from unutbu is enough.

I used selenium with .net not python so I can't give the detailed code of python but for stable you should wait 3 times:

  1. Wait for next page or ajax content loaded, in this case you should wait until text of .CafeF_Paging td span strong is the page number you excepted.
  2. Wait for jquery loaded, until jQuery.active == 0
    check out http://sullerton.com/2013/08/selenium-webdriver-wait-for-ajax-with-python/
  3. Wait for element visible, not only located, use visibility_of_element_located.

Upvotes: 1

Related Questions