CODEWITHSUNDEEP
pythonseleniumscreen-scraping
Luis Ramon Ramirez Rodriguez
Luis Ramon Ramirez Rodriguez

Reputation: 10383

Selenium doesn't load the complete page

I'm trying to scrape this site:

https://www.lobstersnowboards.com/shop/eiki-pro-model--459

And one of the fields I want to get is the available models:

enter image description here

But this part is not getting loaded by selenium:

enter image description here

I had tried with both, firefox and chrome, getting the same result.

Upvotes: 0

Views: 3164

Answers (1)

Tarun Lalwani
Tarun Lalwani

Reputation: 146510

So the issue that it doesn't work is not related to Selenium.

It is because of localization. As you can see in your top bar REST OF THE WORLD and you are not in United States which should show you the US prices

Below script shows how it works

from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://www.lobstersnowboards.com/shop/eiki-pro-model--459")
driver.find_element_by_css_selector("button[title='Rest of the World'] span.caret").click()
driver.find_element_by_xpath("//li[.='United States']").click()
import time
time.sleep(4)
driver.find_element_by_css_selector("div.options [title='Select'] span.caret").click()
elems = driver.find_elements_by_css_selector("div.options ul > li a span")
for elem in elems:
    print(elem.text.strip())

I have used a dirty quick sleep, which you should not but that is just for the demo

Output

And below is the actual page in the chrome

Select Options

Somethings that will impact you later but not in Chrome 65

On January 19, 2017, a public posting to the mozilla.dev.security.policy newsgroup drew attention to a series of questionable website authentication certificates issued by Symantec Corporation’s PKI. Symantec’s PKI business, which operates a series of Certificate Authorities under various brand names, including Thawte, VeriSign, Equifax, GeoTrust, and RapidSSL, had issued numerous certificates that did not comply with the industry-developed CA/Browser Forum Baseline Requirements. During the subsequent investigation, it was revealed that Symantec had entrusted several organizations with the ability to issue certificates without the appropriate or necessary oversight, and had been aware of security deficiencies at these organizations for some time.

From: https://security.googleblog.com/2017/09/chromes-plan-to-distrust-symantec.html

The SSL certificate used to load resources from https://www.lobstersnowboards.com will be distrusted in M66. Once distrusted, users will be prevented from loading these resources. See https://g.co/chrome/symantecpkicerts for more information.

This is the error that is shown in Chrome 65.

Upvotes: 1

Related Questions