CODEWITHSUNDEEP
pythonajaxseleniumweb-scraping
Rahan
Rahan

Reputation: 29

Scraping dynamic dropdown with Selenium in Python

I woud like a bit of help with the following. I am trying to scrape the elements of the tickers' dropdown on this website: https://live.hxro.io/tixwix

My code is as follow using selenium

from selenium import webdriver
from selenium.webdriver.support.select import Select
from selenium.webdriver.chrome.options import Options

url = "https://live.hxro.io/tixwix"
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path = r'C:\geckodriver\chromedriver.exe',options = chrome_options)
driver.get(url)

tickers = driver.find_elements_by_class_name('lastprice-toggle')

tickers[0].text

This will only return

'BTC\nLast Price\n$39,255.07'

As it is an Ajax call I am not sure how to retrieve the other tickers in an efficient way. I thought the function find_element's' will return all the elements into a list but I only get the first one tickers[1] is out of bound.

Screenshot of the page source: enter image description here

Thanks

Upvotes: 1

Views: 229

Answers (2)

Prophet
Prophet

Reputation: 33351

There are several issues here:

  1. You should add
options.add_argument("--start-maximized")
  1. You should add a wait / delay
  2. You should open the drop list
  3. use correct locator to get elements in the drop list
    UPD
    5)In case of ElementClickInterceptedException click it with JavaScript. Not the best practice, but will work.
    I Think the code should look like this:
from selenium import webdriver
from selenium.webdriver.support.select import Select
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 30)

url = "https://live.hxro.io/tixwix"
chrome_options = Options()
chrome_options.add_argument("--headless")
options.add_argument("--start-maximized")
driver = webdriver.Chrome(executable_path = r'C:\geckodriver\chromedriver.exe',options = chrome_options)
driver.get(url)

last_price_toggle = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.lastprice-toggle')))
driver.execute_script("arguments[0].click();", last_price_toggle)

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.lastprice-item')))

tickers = driver.find_elements_by_css_selector('div.lastprice-item span.moon')
for ticker in tickers:
    print(ticker.text)

Upvotes: 1

cruisepandey
cruisepandey

Reputation: 29362

The below code works fine on my local :

Explanation :

You need to click on accept cookies button and shall try with ExplicitWait plus you would need to click on a svg icon which is there in your drop down (XPATH : //span[text()='Last Price']/../following-sibling::*).

In the end span.moon contains all the elements that you are looking for.

Code :

executablePath = r'C:\geckodriver.exe'
options = webdriver.FirefoxOptions()
options.add_argument("--headless")
driver = webdriver.Firefox(executable_path = executablePath, options=options)
driver.maximize_window()
driver.get("https://live.hxro.io/tixwix")
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.ID, "onetrust-accept-btn-handler"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Last Price']/../following-sibling::*"))).click()
sleep(5)
for values in driver.find_elements(By.CSS_SELECTOR, "span.moon"):
    print(values.get_attribute('innerHTML'))

output :

$39,321.02
$39,321.02
$8,728.5
$0.31063
$2,435.337
$24.232
$40.99
$8.9730
$22.685
$37,234.01
 HIGHER
<span class="tooltip-text">CLOSE ABOVE</span>
(CLOSE ABOVE)
(TOUCH)
 $100,000
14.20X
 $100,000
1000.05X
 $68,000
152.69X
 $66,000
127.38X
 $64,000
117.34X
 $63,000
112.25X
 $62,000
105.95X
 $61,000
4.490X
 $60,000
92.80X
 $44,000
4.790X

Process finished with exit code 0

Upvotes: 2

Related Questions