CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scraping
Psyke
Psyke

Reputation: 43

Python Selenium Scraping missing image from url

Hello all my task is to scrap source urls from offer link like this one.

But when I try to get the elements like this (Note that I make 2 requests of the url to get the cookies, because the first time is redirecting me to the main page):

driver = webdriver.Firefox(executable_path="C:\\selenium-drivers\\geckodriver.exe")
driver.get("http://www.kmart.com/joe-boxer-men-s-pajama-shirt-pants-plaid/p-046VA92629712P")
driver.get("http://www.kmart.com/joe-boxer-men-s-pajama-shirt-pants-plaid/p-046VA92629712P")
img_element = driver.find_elements_by_class_name("main-image")

No elements are found and when i try to search them in the source code in the browser with Ctrl+U they are missing. Why is this happening? Is anyone who can tell me how to get these images.

Upvotes: 1

Views: 830

Answers (2)

Hisagr
Hisagr

Reputation: 621

Or you can just find by xpath and then get image url

>>> driver.get('http://www.kmart.com/joe-boxer-men-s-pajama-shirt-pants-plaid/p-046VA92629712P')
>>> s = driver.find_element_by_xpath('//*[@id="overview"]/div[1]/img')
>>> s.get_attribute('src') 
'http://c.shld.net/rpx/i/s/i/spin/-122/prod_2253990712?hei=624&wid=624&op_sharpen=1'

Upvotes: 0

alecxe
alecxe

Reputation: 473763

You just need to tell selenium to be patient and wait for element's visibility:

from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

# driver definition here

driver.get("http://www.kmart.com/joe-boxer-men-s-pajama-shirt-pants-plaid/p-046VA92629712P")

wait = WebDriverWait(driver, 10)

# get the main image element
img_element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'main-image')))
print(img_element.get_attribute("alt"))

driver.close()

For the demonstration purposes, it prints the alt attribute of the image, which is:

Joe Boxer Men's Pajama Shirt & Pants - Plaid

Upvotes: 2

Related Questions