CODEWITHSUNDEEP
pythonseleniumselenium-webdriverfindelement
Miqro
Miqro

Reputation: 477

Selenium can't seem to find and click a button on website? - Python

I am trying to make software that automaticly collects data for me from fiverr.com website.

It searches for specific term, and then it should press LOAD MORE button at the end of page, and keep clicking it until it loads all the 'gigs'.

Anyway, I have good idea of how to make it keep clicking untill everything is loaded, but I cant seem to even click it.

This is my code for selecting the button and clicking it:

driver.find_element_by_xpath('//*[@class="btn-standard-lrg btn-white"]').click()

And I keep getting following error:

Element is not currently visible and so may not be interacted with

If you go to: fiverr gig url and scroll down, you can see the LOAD MORE button.

Upvotes: 1

Views: 4003

Answers (3)

John Stephenson
John Stephenson

Reputation: 501

Looking at the website some jiggery-pokery seems to be going on. From experience, I would suggest:

General:

  1. Don't screen scrape information from 3rd party websites. If they change the website your code will break - perhaps contact the site to see if they have an API?
  2. Always use the PageObject pattern as it'll keep your code DRY (think components rather than 'pages').

Specific:

  1. Try selecting a parent element first and then work down to the element you want. This often leads you to the problem.
  2. If you don't have any automatic retry configured in selenium you could be running that code before the element is made visible. As you can see from the error message, the element is there it's just not visible (I've got selenium experience but very little python so can't help you there :-( ).

Upvotes: 0

alecxe
alecxe

Reputation: 473763

First of all, using "layout-oriented" or "design-oriented" classes like btn-standard-lrg and btn-white is a bad practice.

Instead, there is a convenient locator "by link text", use it:

load_more = driver.find_element_by_link_text("LOAD MORE")

Note how readable and simple it is.

You may also need to wait until the "Load More" button would become visible:

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)
load_more = wait.until(EC.visibility_of_element_located((By.LINK_TEXT, 'LOAD MORE')))

And, here is the complete code that would click "Load More" until it becomes invisible which would mean all posts are loaded:

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException


driver = webdriver.Chrome()  # or webdriver.Firefox()
driver.get("https://www.fiverr.com/search/gigs?utf8=%E2%9C%93&search_in=everywhere&source=top-bar&query=explainer%20video&page=1&layout=lists&filter=new")

wait = WebDriverWait(driver, 10)
while True:
    try:
        load_more = wait.until(EC.visibility_of_element_located((By.LINK_TEXT, 'LOAD MORE')))
    except TimeoutException:
        break

    load_more.click()

for item in driver.find_elements_by_css_selector("div.gig-item h3 a.gig-link-main"):
    print item.text.strip()

Upvotes: 4

Pippo
Pippo

Reputation: 925

This seems to be an issue given you are trying to interact with an element that is not visible in the page (you need to scroll down to the bottom of the page to visualise it)...

Try doing something like this:

element = driver.find_element_by_xpath('//*[@class="btn-standard-lrg btn-white"]')

location = element.native.location[:y]

execute_script("window.scrollTo(0, #{location})")

element.click

or simply prior to clicking scroll to the bottom of the page:

execute_script("window.scrollTo(0, 10000)")

Upvotes: 0

Related Questions