CODEWITHSUNDEEP
pythonseleniumweb-scraping
AndreasInfo
AndreasInfo

Reputation: 1227

Capture elements with Selenium which are loaded lazy in Python

Trying to click a button with Selenium, but I keep getting an error:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"AGREE"}

Here's the button I am trying to click.

enter image description here

I assume, the popup is loaded lazy. I found some sources, but could not make it work. Here's my code

import pandas as pd
import bs4
import selenium as sel
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import time
import os

driver = sel.webdriver.Chrome() #uses Chrome driver in usr/bin/ from https://chromedriver.chromium.org/downloads
url = 'https://fbref.com/en/squads/0cdc4311/Augsburg'
driver.get(url)
time.sleep(5)
html = driver.page_source
soup = bs4.BeautifulSoup(html, 'html.parser')

#click button -> accept cookies
element = driver.find_element(By.LINK_TEXT, "AGREE")
element.click()

>>> NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"AGREE"}

I also tried

[...]
driver = sel.webdriver.Chrome() #uses Chrome driver in usr/bin/ from https://chromedriver.chromium.org/downloads
driver.implicitly_wait(10) # seconds'
[...]

and the popup is definitively there. But still get the same error.

Upvotes: 0

Views: 296

Answers (1)

HedgeHog
HedgeHog

Reputation: 25048

What happens?

You try to select a <button> via .LINK_TEXT, "AGREE", what won't work, cause it is a <button> not a link.

How to fix?

Wait for the <button> selected by xpath to be clickable:

#click button -> accept cookies
element = WebDriverWait(driver, 3).until(EC.element_to_be_clickable((By.XPATH, '//button[text()="AGREE"]')))
element.click()

Upvotes: 1

Related Questions