CODEWITHSUNDEEP
pythonhtmlcssselenium
anarchy
anarchy

Reputation: 5184

How to get values from search suggestions after keying in text using python selenium?

When you enter something for example apple into the search bar at https://finance.yahoo.com/ there is a search suggestions menu that appears.

enter image description here

I am trying to get it to return a list, dictionary or dataframe of the values in that drop down box.

For example 

{'AAPL':['Apple Inc.','Equity - NMS','https://finance.yahoo.com/quote/AAPL?p=AAPL&.tsrc=fin-srch'],
 'AAPL.BA':['Apple Inc.','Equity - BUE','https://finance.yahoo.com/quote/AAPL.BA?p=AAPL.BA&.tsrc=fin-srch'],
  .....}

or

['AAPL','Apple Inc.','Equity - NMS','https://finance.yahoo.com/quote/AAPL?p=AAPL&.tsrc=fin-srch']
['APPL.BA','Apple Inc.','Equity - BUE','https://finance.yahoo.com/quote/AAPL.BA?p=AAPL.BA&.tsrc=fin-srch']

The last value is the hyperlink from clicking the link.

Here is my code so far,

options = Options()

driver = webdriver.Chrome(executable_path=r'C:\Program Files\chromedriver\chromedriver.exe',options=options)

url = "https://finance.yahoo.com/"
driver.get(url)
time.sleep(2)

inputElement = driver.find_element_by_xpath('//*[@id="yfin-usr-qry"]')
inputElement.send_keys('apple')
time.sleep(2)

web_elem_list = driver.find_elements_by_xpath(".//ul[@class='M(0)']/li/div/div")
suggests = [web_elem.text for web_elem in web_elem_list]

print(suggests)

driver.close()

But the output keeps coming up empty, I cant seem to locate the elements in the suggestion box.

I also tried using web_elem_list = driver.find_elements_by_xpath(".//ul[@class='f470fc71']/li/div/div")

But it doesnt have any values.

How do I,

  1. Find the xpath of the suggestion box?
  2. Create a dataframe, dictionary or list of all the results (including the hyperlink).

UPDATE:

I have figured out the first part of the question, the xpath had one too many /div. I updated my question and the part of the code works now.

But I still haven't figured out the second part of the question, I still cant get the "Equity - NMS" part and hyperlinks.

Upvotes: 1

Views: 378

Answers (2)

0buz
0buz

Reputation: 3503

Made some changes to your script around waiting and xpaths. The result will be the suggested data in a pandas dataframe.

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import time
import pandas as pd

options=webdriver.ChromeOptions()
options.add_argument('start-maximized')

driver = webdriver.Chrome(options=options)
url = "https://finance.yahoo.com/"
driver.get(url)

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="yfin-usr-qry"]'))).send_keys('apple')

WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div//*[contains(text(),'Symbols')]")))
web_elem_list = driver.find_elements_by_xpath(".//div[@data-test='search-assist-input-sugglst']/div/ul[1]/li/div")
results = pd.DataFrame()

for web_elem in web_elem_list:
    suggests=[]

    code=web_elem.find_element_by_xpath("./div/div").text
    suggests.append(code)

    name=web_elem.find_element_by_xpath("./div/div/following-sibling::div").text
    suggests.append(name)

    equity=web_elem.find_element_by_xpath("./div/following-sibling::div").text
    suggests.append(equity)

    hyperlink=f'https://finance.yahoo.com/quote/{code}?p={code}&.tsrc=fin-srch'
    suggests.append(hyperlink)

    results=results.append(pd.Series(suggests), ignore_index=True)

print(results)

driver.close()

Output: enter image description here

Upvotes: 2

muhmann
muhmann

Reputation: 201

This worked for me:

driver.find_elements_by_css_selector("ul[class=f470fc71] li[role=link][data-type=quotes]")
suggestions = [web_elem.get_attribute("title") for web_elem in web_elem_list]

output:

['Apple Inc.', 'Apple Hospitality REIT, Inc.', 'Apple Inc.', 'Apple Rush Company, Inc.', 'Apple Inc.', 'ApplePie Capital']

Upvotes: 0

Related Questions