Reputation: 886
I am trying to scrape a table with multiple pages. The next page is obtained by clicking on the 'Next Page button' (See code snippet).
<a class="botons" id="btn2" href="javascript:void(0)">
Next Page
<i class="fas fa-long-arrow-alt-right"></i>
</a>
Selenium finds the "button" and has no trouble "clicking" via the following code:
btn_next = self.browser.find_element_by_partial_link_text("Next Page")
btn_next.click()
However, the page just refreshes and the table doesn't update to its next page.
Any clues to what's going wrong here?
Edit: table can be found at https://www.proxy-list.download/HTTPS
Edit2:
chrome_options = Options()
chrome_options.add_argument("--enable-javascript")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--headless")
Upvotes: 0
Views: 251
Reputation: 193308
The desired element is an JavaScript enabled element so to locate and click()
on the element you have to induce WebDriverWait for the element_to_be_clickable()
and you can use either of the following Locator Strategies:
Using CSS_SELECTOR
:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.botons[id^='btn2'] i.fas.fa-long-arrow-alt-right"))).click()
Using XPATH
:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='botons' and normalize-space()='Next Page']/i[@class='fas fa-long-arrow-alt-right']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Upvotes: 0
Reputation: 29382
There is one id assigned to that button btn2
and it's unique too.
You should give preference to id over link Text.
That said, Next Page link isn't present in view point.For that first you have to move the focus of driver like this :
wait = WebDriverWait(self.browser,10)
next_page = wait.until(EC.visibility_of_element_located((By.ID, "btn2")))
ActionChains(self.browser).move_to_element(next_page).perform()
next_page.click()
Imports :
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Upvotes: 1