CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scrapingscrapy
Pravesh Jain
Pravesh Jain

Reputation: 4288

Get JavaScript function call value using Selenium

I am scraping web pages using python-scrapy which works pretty well for static content. I am trying to scrape a url from this page but as it turns out, it is returned through a javascript call. For this I am using selenium but unable to figure out how to do it.

If you click on the "size chart" on the given link, you see a pop up opening mentioning the size guide. How can I get the url of this guide in my program?

I am also facing a similar problem on koovs as well getting the size guide. If anyone could guide on any of the links, I'd be really grateful.

Upvotes: 0

Views: 218

Answers (1)

alecxe
alecxe

Reputation: 473843

Locate the "size chart" link by link text, click it and extract the data, example:

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Firefox()
driver.get('http://www.jabong.com/athena-Red-Black-Top-476472.html?pos=3')

wait = WebDriverWait(driver, 10)
chart = wait.until(EC.presence_of_element_located((By.LINK_TEXT, "size chart")))
chart.click()

for title in driver.find_elements_by_css_selector("div.size-chart-body div.size-chart table th"):
    print title.text

driver.close()

Prints (table header row, for the sake of an example):

Indian Size
Euro Size
Garment Bust (In.)
Garment Waist (in.)
Garment Hip (in.):

Note that you don't need selenium to get the size chart data, it is already inside the DOM, but invisible until you click "size chart". You can reach the same size chart table with Scrapy. Demo from the "Scrapy Shell":

$ scrapy shell http://www.jabong.com/athena-Red-Black-Top-476472.html?pos=3
In [1]: for title in response.css("div.size-chart-body div.size-chart table th")[1:]:
    print title.xpath("text()").extract()[0]
   ...:     
Indian Size
Euro Size
Garment Bust (In.)
Garment Waist (in.)
Garment Hip (in.)

In case of Koovs, you can still avoid using selenium and construct the size chart URL manually extracting the category and deal name, e.g.:

$ scrapy shell http://www.koovs.com/only-onlall-stripe-ls-shirt-59554.html?from=category-651
In [1]: category = response.xpath("//input[@id='master_category_name_id_ref']/@value").extract()[0]

In [2]: deal = response.xpath("//input[@id='deal_id']/@value").extract()[0]
In [3]: "http://www.koovs.com/koovs/sizechart/women/{category}/{deal}".format(category=category, deal=deal)
Out[3]: 'http://www.koovs.com/koovs/sizechart/women/Shirts--651--799--896/59554'

And, if you still want to go with selenium, here you are:

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Firefox()
driver.get('http://www.koovs.com/only-onlall-stripe-ls-shirt-59554.html?from=category-651&skuid=236376')

wait = WebDriverWait(driver, 10)
chart = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "a[size_chart]")))
chart.click()

driver.switch_to.window(driver.window_handles[-1])

print driver.current_url

driver.close()

Prints:

http://www.koovs.com/koovs/sizechart/women/Shirts--651--799--896/59554

Upvotes: 1

Related Questions