CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scraping
adominey
adominey

Reputation: 102

Most of HTML elements are empty while checking with selenium

I'm learning web scraping with selenium. Inspect shows that element that I'm searching has class name 'movie-link'. Here's my code:

def get_movies(driver, name, scroll_num):
    elem = driver.find_element_by_class_name("form-control")
    elem.clear()
    elem.send_keys(name)
    elem.send_keys(Keys.RETURN)

    scroll_down(driver, scroll_num)

    return map(lambda x: x.get_attribute('href'), driver.find_elements_by_class_name('movie-link'))

driver.find_elements_by_class_name('movie-link') could not find anything, so I checked (with selenium) the element which contains 'movie-link' and its innerHTML is 'placeholder'.

Is this some kind of protection or am I forgetting something?

EDIT: I'm trying to parse http://imovies.cc

Upvotes: 0

Views: 304

Answers (1)

Sers
Sers

Reputation: 12255

Explicit wait:

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

movies = WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".movie-link"))

Upvotes: 2

Related Questions