CODEWITHSUNDEEP
pythonhtml
Lachlan Moore
Lachlan Moore

Reputation: 53

Python Selenium divs hidden in inspect, cannot see button to show divs on actual page

I am writing a web scraping script in Python using Selenium (link here https://www.hltv.org/stats/players) where I want to scrape all the links associated for each Player.

It shows a lot of players but not all of them and I cannot see a button to show these elements on the actual page, they are hidden away behind the page. If I inspect the page there is a table that shows some which is what is seen on the page, then there is more under the class name "hidden". But then if I scroll to the bottom of inspect there is a button that says for me to click to show the other 2000 players.

I am able to scrape all the players links except the other 2000 nodes that needs the button click in inspect. I have not found anything online where people have a similar problem, maybe there is that I have not found.

I am using the Google Chrome browser with Python 3.8.1 in VS Code.

How would someone go about doing this? Here is my current code:

from selenium import webdriver

driver = webdriver.Chrome(r'D:\chromedriver_win32\chromedriver')
driver.get(r"https://www.hltv.org/stats/players")

a_elems = driver.find_elements_by_class_name("context-button")

for elem in a_elems:
    print(elem.get_attribute("href"))

Thanks

Upvotes: 0

Views: 27

Answers (1)

Agnia
Agnia

Reputation: 84

a_elems = driver.find_elements_by_css_selector("a[href*='players']")

for elem in a_elems:
  print(elem.get_attribute("href"))

Here I took all links which contained "players" in link.

Upvotes: 1

Related Questions