CODEWITHSUNDEEP
pythonseleniumweb-scraping
Murali Kumar
Murali Kumar

Reputation: 47

Selenium - response data does not contain certain elements

I'm trying to scrape this page

https://www.tribeofdumo.com/product-page/gbemi-dress-purple

and in particular I'm trying to get the list of sizes in the dropdown menu by accessing the element <div class ='option-selection-item'>.

I can see this element when doing an inspect element, however in the response data from Selenium I am unable to locate this element. So how do I go about extracting the data from the sizes dropdown?

Here is my code:

from time import sleep
import re
from scrapy.selector import Selector
from selenium import webdriver

driver = webdriver.Firefox()
driver.get("https://www.tribeofdumo.com/product-page/gbemi-dress-purple")
sleep(10)
driver.switch_to.frame("TPAMultiSection_jc31t3yyiframe")
sleep(10)
html = driver.execute_script("return document.documentElement.outerHTML;")

Upvotes: 0

Views: 49

Answers (1)

SIM
SIM

Reputation: 22440

I'm not sure whether you meant to do the following. However, It produces the different size available in that dropdown:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.tribeofdumo.com/product-page/gbemi-dress-purple")
driver.switch_to.frame(driver.find_element_by_id("TPAMultiSection_jc31t3yyiframe"))
driver.find_element_by_css_selector(".select2-arrow").click()
for item in driver.find_elements_by_css_selector(".ui-select-container .option-selection-item"):
    print(item.text)

driver.quit()

Output:

S
M
L
XL
2XL

Upvotes: 2

Related Questions