CODEWITHSUNDEEP
pythonseleniumweb-scraping
Luka Kunej
Luka Kunej

Reputation: 35

Selenium driver not returning all web elements even though they are the same class

I'm trying to make a web scraper for a webpage using selenium.

the webpage is https://www.supersport.hr/sport/dan/0/sport/1

I am able to get most of the web elements I'm trying to get but my script doesn't return all of them even though they are the same class.

page example: enter image description here

In this case my script returns all of the above divs, but cuts off at "SRL GRČKA 1." and doesn't get any leagues below.

the HTML markup for each of these leagues are the same:

enter image description here

I'm getting a list of these elements in python like this:

football_leagues_elements = driver.find_elements_by_css_selector("div.sportska-liga.nogomet")

I've also tried with this code but it returns the same result:

football_leagues_elements = driver.find_elements_by_xpath("//div[contains(@class, 'sportska-liga-wrap')]//div[contains(@class, 'nogomet')]")

I think all the leagues are loaded to the page at the same time.

My question is why are some divs not included in the webelement list?

Any help is welcome.

Upvotes: 0

Views: 1455

Answers (1)

KunduK
KunduK

Reputation: 33384

The Website is NOT accessible here.However try Induce Explicit wait and wait for all element visible.

football_leagues_elements=WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.sportska-liga.nogomet")))

You need to import following libraries.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

If above doesn't work then try to scroll the page first and then check for elements.

#Scroll bottom of the page.
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
#Get the elements.
football_leagues_elements=WebDriverWait(driver,20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"div.sportska-liga.nogomet")))

Upvotes: 3

Related Questions