CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriverweb-scraping
SIM
SIM

Reputation: 22440

Trouble scraping titles from a webpage

I've written a script in python with selenium to parse some results populated upon filling in an inputbox and preessing the Go button. My script does this portion well at this moment. However, my main goal is to parse the title of that container visible as Toys & Games as well.

This is my try so far (I could not find any idea to make a loop to do the same for all the containers):

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

url = "https://www.fbatoolkit.com/"

driver = webdriver.Chrome()
driver.get(url)
time.sleep(3)
driver.find_element_by_css_selector(".estimator-container .estimator-input").send_keys("25000",Keys.RETURN)
time.sleep(2)
item = driver.find_element_by_css_selector(".estimator-result div").text
print(item)
driver.quit()

The result I get:

4 (30 Days Avg)

Result I would like to have:

Toys & Games
4 (30 Days Avg)

Link to an image in which you can see how they look like in that site. Expected fields are also marked with a pencil to let you know the location of the fields I'm trying to parse.

Upvotes: 3

Views: 58

Answers (1)

Andersson
Andersson

Reputation: 52665

Try below code to get required output

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC


url = "https://www.fbatoolkit.com/"

driver = webdriver.Chrome()
driver.get(url)

for container in wait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div[class='chart-container']"))):
    wait(container, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.estimator-input"))).send_keys("25000", Keys.RETURN)
    title = wait(container, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".chart text"))).text
    item = wait(container, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".estimator-result div"))).text
    print(title, item)

driver.quit()

Upvotes: 3

Related Questions