CODEWITHSUNDEEP
pythonseleniumselenium-chromedriver
kamo758
kamo758

Reputation: 23

Scraping with selenium, can not get a button to click and load a hidden table with the data i want

I am trying to scrape a form that has nav tabs that display 4 different ways to search through the requested information. The URL loads into the tab that I want to search from and with the date fields automatically filled in. When a human user wants to look through the info they only have to click the search button that is in the selected tab, which enabled a loading screen and fills and enables a table below with the data I want to scrape. I am using python 3 and selenium currently.

Button code : 

<button type="submit" id="community-search" class="btn btn-primary" data-bind="click: submitSearch, enable: searchEnabled">Search</button>

This is my code right now: 

#start of code after imports
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')

dir_path = os.path.dirname(os.path.realpath(__file__))
chromedriver = dir_path + "/chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
browser = webdriver.Chrome(chrome_options = options, executable_path = chromedriver)

# LOGS INTO THE SITE 
browser.get('https://***.com/')
username = browser.find_element_by_name('UserName')
username.send_keys('***') 
password = browser.find_element_by_name('Password')
password.send_keys('***')
browser.find_element_by_css_selector('#sign-in').click()
# LOGIN WORKS

#GOES TO THE SCHEDUALS PAGE 
browser.get('https://***.com/Schedules')
#I KNOW IT IS NAVIGATING TO THIS PAGE

# I HAVE TRIED WAITING AND FOUND NO LUCK, TRYING TO CLICK THE BUTTON LEADS TO A HUGE LIST OF TRACEBACKS
browser.find_element_by_css_selector('community-search').click()
#WebDriverWait(browser, 5).until(EC.presence_of_element_located((By.ID, 'community-search'))).click

#end of code

Errors- 

    Traceback (most recent call last):
  File "Project.py", line 29, in <module>
    browser.find_element_by_css_selector('community-search').click()
  File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 589, in find_element_by_css_selector
    return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
  File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 955, in find_element
    'value': value})['value']
  File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"community-search"}

I am fairly new to python and web scraping, this is my first post so I hope this was enough info. Any help would be very much appreciated

ImageOfSiteBeforeBtnPress ImageOfSiteAfterBtnPress

Upvotes: 2

Views: 363

Answers (2)

undetected Selenium
undetected Selenium

Reputation: 193088

The error says it all :

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"community-search"}

The error clearly says NoSuchElementException, as the css-selector which you have used as follows doesn't identifies a unique element :

browser.find_element_by_css_selector('community-search').click()

Incase community-search is the value of the id, a proper css_selector would be (adding a # before the value) as follows :

browser.find_element_by_css_selector("#community-search").click()

But as you are trying to search the WebElement soon after invoking browser.get() you may need to induce WebDriverWait in-conjunction with expected_conditions clause as element_to_be_clickable as follows :

WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#community-search"))).click()

Incase you want to use the id attribute you can use the following line of code as follows :

browser.find_element_by_id("community-search").click()

To click on the WebElement using the id attribute along with WebDriverWait :

WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.ID, "community-search"))).click()

Upvotes: 0

Filipe Freire
Filipe Freire

Reputation: 833

Your CSS selector is missing #since community-search is an id.

So instead of:

browser.find_element_by_css_selector('community-search').click()

It should be:

browser.find_element_by_css_selector('#community-search').click()

Another way to do it is:

browser.find_element_by_id('community-search').click()

Upvotes: 1

Related Questions