CODEWITHSUNDEEP
pythonhtmlseleniumweb-scraping
alecbeau
alecbeau

Reputation: 41

Selenium not downloading image from web

Website I am trying to download images from: https://db.ygoprodeck.com/search/?&num=30&offset=0&view=List

I am trying to get an image of the yugioh card and store that image in a database

This is the code I am currently running:

import selenium
from selenium import webdriver
DRIVER_PATH = 'my_path'
wd = webdriver.Chrome(executable_path = DRIVER_PATH)
wd.get('https://db.ygoprodeck.com/search/?&num=30&offset=120&view=List')
images = wd.find_elements_by_xpath("//img[@class='lazy']")
if(images):
    print("True")
wd.close()

Upvotes: 1

Views: 195

Answers (2)

xaander1
xaander1

Reputation: 1160

The reason you cant get the images its because they are lazy loaded. You have to wait until they are loaded.

Selenium solves this problem by two types of waiting explicit waiting or implicit waiting the more extreme case would be using time.sleep()

Read More Here documentation

Solution

Using Implicit Wait

...
wd.get('https://db.ygoprodeck.com/search/?&num=30&offset=120&view=List')
wd.implicitly_wait(10)
images = wd.find_elements_by_xpath("//img[@class='lazy']")
....

Using Explicit Wait

...
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
...
images = WebDriverWait(wd, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "img.lazy")))
...

A more fast solution would be using requests, urllib3 or scrapy as @bigbounty suggested getting data directly from the ajax call.

Upvotes: 0

bigbounty
bigbounty

Reputation: 17358

When you analyse the website, the website makes an ajax call to load all the data.

import requests, json

res = requests.get("https://db.ygoprodeck.com/api_internal/v7/cardinfo.php?&num=30&offset=0&view=List&misc=yes")

print(res.json())

Output:

{'data': [{'id': 34541863,
   'name': '"A" Cell Breeding Device',
   'type': 'Spell Card',
   'desc': 'During each of your Standby Phases, put 1 A-Counter on 1 face-up monster your opponent controls.',
   'race': 'Continuous',
   'archetype': 'Alien',
   'card_sets': [{'set_name': 'Force of the Breaker',
     'set_code': 'FOTB-EN043',
     'set_rarity': 'Common',
     'set_rarity_code': '(C)',
     'set_price': '1.03'}],
   'card_images': [{'id': 34541863,
     'image_url': 'https://storage.googleapis.com/ygoprodeck.com/pics/34541863.jpg',
     'image_url_small': 'https://storage.googleapis.com/ygoprodeck.com/pics_small/34541863.jpg'}],
   'card_prices': [{'cardmarket_price': '0.11',
     'tcgplayer_price': '0.22',
     'ebay_price': '2.25',
     'amazon_price': '0.25',
     'coolstuffinc_price': '0.25'}],
   'misc_info': [{'beta_name': 'A Cell Breeding Device',
     'views': 202029,
     'viewsweek': 5270,
     'upvotes': 34,
     'downvotes': 26,
     'formats': ['Duel Links', 'TCG', 'OCG'],
     'tcg_date': '2007-05-16',
     'ocg_date': '2007-02-15'}]},
  {'id': 64163367,
   'name': '"A" Cell Incubator',
   'type': 'Spell Card',
   'desc': 'Each time an A-Counter(s) is removed from play by a card effect, place 1 A-Counter on this card. When this card is destroyed, distribute the A-Counters on this card among face-up monsters.',
   'race': 'Continuous',
   'archetype': 'Alien',
   'card_sets': [{'set_name': "Gladiator's Assault",
     'set_code': 'GLAS-EN062',
     'set_rarity': 'Common',
     'set_rarity_code': '(C)',
     'set_price': '1'}],
   'card_images': [{'id': 64163367,
     'image_url': 'https://storage.googleapis.com/ygoprodeck.com/pics/64163367.jpg',
     'image_url_small': 'https://storage.googleapis.com/ygoprodeck.com/pics_small/64163367.jpg'}],
   'card_prices': [{'cardmarket_price': '0.10',
     'tcgplayer_price': '0.26',
     'ebay_price': '1.25',
     'amazon_price': '0.25',
     'coolstuffinc_price': '0.25'}],
   'misc_info': [{'beta_name': 'A Cell Incubator',
     'views': 165264,
     'viewsweek': 3644,
     'upvotes': 11,
     'downvotes': 11,
     'formats': ['Duel Links', 'TCG', 'OCG'],
     'tcg_date': '2007-11-14',
     'ocg_date': '2007-07-21'}]},
  {'id': 91231901,
   'name': '"A" Cell Recombination Device',
   'type': 'Spell Card',
   'desc': 'Target 1 face-up monster on the field; send 1 "Alien" monster from your Deck to the Graveyard, and if you do, place A-Counters on that monster equal to the Level of the sent monster. During your Main Phase, except the turn this card was sent to the Graveyard: You can banish this card from your Graveyard; add 1 "Alien" monster from your Deck to your hand.',
   'race': 'Quick-Play',
   'archetype': 'Alien',
   'card_sets': [{'set_name': 'Invasion: Vengeance',
     'set_code': 'INOV-EN063',
     'set_rarity': 'Common',
     'set_rarity_code': '(C)',
     'set_price': '0.92'}],
   'card_images': [{'id': 91231901,
     'image_url': 'https://storage.googleapis.com/ygoprodeck.com/pics/91231901.jpg',
...
...
...
..

The json data contains all the data including image links

Upvotes: 1

Related Questions