CODEWITHSUNDEEP
pythonseleniumweb-scraping
user3788728
user3788728

Reputation: 59

python dynamic webscraping javascript content

I am using Python and Selenium to scrape a website. What I do is go to the homepage, type in a keyword, such as 1300746-79-5. On the resulting page, I am trying to scrape the data in the "pricing" section. Specifically, I need to get the "SKU-Pack Size" and "Price(USD)" information. But these information is Javascript encripted, so I cannot see them in the source code. I am wondering how I can achieve this.

I have written some code that gets me to the page of interest, but I still cannot see the javascript information. Here is what I have so far.

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pprint

# Create a new instance of the Firefox driver
driver = webdriver.Chrome('C:\Users\Rei\Desktop\chromedriver.exe')
driver.get("http://www.sigmaaldrich.com/united-states.html")

print driver.title
inputElement = driver.find_element_by_name("Query")

# type in the search
inputElement.send_keys("1300746-79-5")
inputElement.submit()

Upvotes: 1

Views: 895

Answers (1)

Yi Zeng
Yi Zeng

Reputation: 32855

Everything you have done looks correct to me.

"SKU-Pack Size" and "Price(USD)" information are not "encrypted", but retrieved after JavaScript clicking action. All you need to do is to click product name or pricing link.

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pprint

driver = webdriver.Chrome()
driver.get("http://www.sigmaaldrich.com/united-states.html")

print driver.title
inputElement = driver.find_element_by_name("Query")

# type in the search
inputElement.send_keys("1300746-79-5")
inputElement.submit()

pricing_link = driver.find_element_by_css_selector("li.priceValue a")
print pricing_link.text
pricing_link.click()

# then deal with the data you want
price_table = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".priceAvailContainer tbody"))
)
print 'price_table.text: ' + price_table.text

driver.quit()

Upvotes: 1

Related Questions