CODEWITHSUNDEEP
pythonselenium
Chris
Chris

Reputation: 5834

How to scrape the pictures from a website utilizing Python Selenium?

The following python code generates the following html page source code from mcmaster.com

from selenium import webdriver 
driver = webdriver.Firefox()
driver.get("https://www.mcmaster.com")
print driver.page_source

However the images are no where to be found in the html. After reading through this Stackoverflow question (Selenium-and-iframe-in-html), I can pull the iframe utilizing the following code 

driver.switch_to.frame(driver.find_element_by_id("ResultsIFrame"))
print driver.page_source
#>> <html><head></head><body></body></html>
driver.switch_to.frame(driver.find_element_by_id("MainIFrame"))
#>> <html><head></head><body></body></html>

Is there any other way to get the picture and/or picture attributes from this site?

I'm using this site as an example case

Upvotes: 0

Views: 170

Answers (2)

undetected Selenium
undetected Selenium

Reputation: 193338

To get the picture attributes from this site you have to induce a waiter through WebDriverWait and you can use the following code block :

from selenium import webdriver 
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome(executable_path=r'C:\WebDrivers\ChromeDriver\chromedriver_win32\chromedriver.exe')
driver.get("https://www.mcmaster.com")
items = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='catg Fastening-Joining']//ul//li//a")))
for i in items :
    print("Item %s link is %s." % (i.text, i.get_attribute("href")))

Console Output :

Item Screws & Bolts link is https://www.mcmaster.com/#Screws.
Item Threaded Rods & Studs link is https://www.mcmaster.com/#Threaded-Rods.
Item Eyebolts link is https://www.mcmaster.com/#Eyebolts.
Item U-Bolts link is https://www.mcmaster.com/#U-Bolts.
Item Nuts link is https://www.mcmaster.com/#Nuts.
Item Washers link is https://www.mcmaster.com/#Standard-Washers.
Item Shims link is https://www.mcmaster.com/#Shims.
Item Helical & Threaded Inserts link is https://www.mcmaster.com/#Threaded-Inserts.
Item Spacers & Standoffs link is https://www.mcmaster.com/#Spacers.
Item Pins link is https://www.mcmaster.com/#Pins.
Item Anchors link is https://www.mcmaster.com/#Standard-Anchors.
Item Nails link is https://www.mcmaster.com/#Nails.
Item Nailers link is https://www.mcmaster.com/#Nailers.
Item Rivets link is https://www.mcmaster.com/#Rivets.
Item Rivet Tools link is https://www.mcmaster.com/#Rivet-Installation-Tools.
Item Staples link is https://www.mcmaster.com/#Staples.
Item Staplers link is https://www.mcmaster.com/#Staplers.
Item Key Stock link is https://www.mcmaster.com/#Machine-Keys.
Item Retaining Rings link is https://www.mcmaster.com/#Retaining-Rings.
Item Cable Ties link is https://www.mcmaster.com/#Cable-Ties.
Item Lanyards link is https://www.mcmaster.com/#Lanyards.
Item Magnets link is https://www.mcmaster.com/#Magnets.
Item Adhesives link is https://www.mcmaster.com/#Adhesives.
Item Tape link is https://www.mcmaster.com/#Fastening-Tape.
Item Hook & Loop link is https://www.mcmaster.com/#Hook-and-Loop.
Item Electrodes & Wire link is https://www.mcmaster.com/#Standard-Welding-Electrodes.
Item Welders link is https://www.mcmaster.com/#Welders.
Item Gas Regulators link is https://www.mcmaster.com/#Welding-Gas-Regulators.
Item Welding Gloves link is https://www.mcmaster.com/#Welding-Gloves.
Item Welding Helmets & Glasses link is https://www.mcmaster.com/#Welding-Eye-Protectors.
Item Protective Screens link is https://www.mcmaster.com/#Protective-Screens.
Item Brazing Alloys link is https://www.mcmaster.com/#Brazing-Supplies.
Item Torches link is https://www.mcmaster.com/#Torches.
Item Solder link is https://www.mcmaster.com/#Solder.
Item Soldering Irons link is https://www.mcmaster.com/#Soldering-Irons.
Item Melting Pots link is https://www.mcmaster.com/#Melting-Pots.

Upvotes: 1

angbargas
angbargas

Reputation: 54

I don't use selenium that much but I scrape by diving deeper into the code. As for the images, I was able to find them here, where I found this sample. You should be able to find everything there. If you have similar things you want to scrape, you can do the same thing I did.

It seems that there is one big image for each category and then just divided through a js code to be placed in the links so you might need a code for that as well if you want the individual images (not 100% sure since I just glanced at the code).

Upvotes: 0

Related Questions