How can you download all non-obvious images with Beautifulsoup/Selenium?

Question

I'm making next little project to learn - it's what I'm trying to do past few days, without success. I want to make list of opals, their prices...and download their images from website. At end (probably) here are two ways: assign opals to images (in word or excel) or just save images with name of opal+price. I succeed at making list of name+price (using help I gained from previous topics), but I can't get into images. They don't have .jpg, here is not 'src' and when I 'extracted' example img link, it was wrong, too. Take a look:

/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F6364%2FF352%2FFF36%2F99BC%2F5F60%2F0A0C%2F6D0B%2FF3AC%2FIMG-8248VOLL_5_ECK.JPG&shop=80300026&width={width}&height=2560

EDIT: 'code edited, too' - I managed to extract just the list of img list I needed.

Below is how img class of this website looks like:

And my code is below - it doesn't download images yet :(

import requests
from bs4 import BeautifulSoup

URL = 'https://www.koroit-opal-company.com/en/c/solid-opals'
page = requests.get(URL)
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"}
soup = BeautifulSoup(requests.get(f'https://www.koroit-opal-company.com/en/c/solid-opals', headers=headers).text, "lxml")
# print(soup.prettify())

names = [n.getText(strip=True) for n in soup.select("div div div a h2")]
# print(names)
prices = [n.getText(strip=True) for n in soup.select("div div div h3")]
# print(prices)

for name, price in zip(names, prices):
    print(f"{name} {price}")

opal_list = soup.find('div', attrs = {'class':'content'})    #gives just fragment of website where opal imgs are
imgs = opal_list.find_all('img')
print(imgs)

example = imgs[0]
x = example.attrs['data-src']
print(x)

result = soup.find_all(lambda tag: tag.name == 'img' and
                       tag.get('class') == ['product-item-image'])

print(result)

Answer 1

you can use the API and all downloaded images ll be next to the executable file:

import requests


def get_info(page: int):
    url = f"https://www.koroit-opal-company.com/api/v2/products?sort=position-asc&resultsPerPage=12&page={page}&categoryId=551DDBD9-8AD4-229C-164A-C0A82AB9D825&locale=en_GB&shop=80300026"
    response = requests.get(url)
    for opal in response.json()['products']:
        img_link = 'https://www.koroit-opal-company.com' + opal['image']['url']
        print(opal['name'], opal['price']['formatted'], img_link)
        download_image(img_link, opal['name'])


def download_image(image_url: str, image_name: str):
    img_data = requests.get(image_url).content
    with open(image_name + '.jpg', 'wb') as handler:
        handler.write(img_data)


get_info(1)

OTPUT for page 1:

1,38 ct Boulder opal 70.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F6364%2FF352%2FFF36%2F99BC%2F5F60%2F0A0C%2F6D0B%2FF3AC%2FIMG-8248VOLL_5_ECK.JPG&shop=80300026
4,27 ct Black opal 3,300.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F5A8D%2F6108%2F2341%2FEB00%2F2803%2F0A0C%2F6D04%2FCDD2%2FIMG-7618VOLL_5_ECK.JPG&shop=80300026
4,52 ct Boulder opal 80.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F6364%2FF4E2%2FAFCE%2FF6ED%2F688E%2F0A0C%2F6D0F%2F3E17%2FIMG-8110VOLL_5_ECK.JPG&shop=80300026
0,85 ct Boulder opal 70.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F6364%2FF540%2FD8D2%2FE5C6%2F9854%2F0A0C%2F6D0B%2F05BA%2FIMG-8018VOLL_5_ECK.JPG&shop=80300026
13,14 ct Crystal opal 400.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F6286%2F56CC%2FED12%2F1E08%2FECEF%2F0A0C%2F6D0B%2F5AAC%2FIMG-9213_HOCH_MAI_22.JPG&shop=80300026
5,85 ct Boulder opal 80.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F6364%2FF59C%2F77B0%2FF735%2F05AF%2F0A0C%2F6D0F%2F8E4C%2FIMG-7990VOLL_5_ECK.JPG&shop=80300026
11,38 ct Boulder opal 2,400.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F594A%2F94B3%2FC880%2FF5AB%2F848B%2FC0A8%2F2AB9%2F4A53%2FIMG-8598VOLL_5_ECK.JPG&shop=80300026
2,41 ct Crystal opal 60.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F6364%2FF66A%2F867E%2F23D7%2FBF76%2F0A0C%2F6D0B%2F947B%2FIMG-8994VOLL_5_ECK.JPG&shop=80300026
6,94 ct Boulder opal 1,600.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F5AB9%2F3AB7%2F8F7A%2F659C%2FDDE4%2F0A0C%2F6D00%2FF73C%2FIMG-7672VOLL_5_ECK.JPG&shop=80300026
2,28 ct Crystal opal 70.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F6364%2FF4D7%2FAEFE%2FF74C%2F9A52%2F0A0C%2F6D0B%2F0587%2FIMG-8045VOLL_5_ECK.JPG&shop=80300026
1,48 ct Crystal opal 70.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F6364%2FF3B9%2F7E20%2F2E27%2FFB71%2F0A0C%2F6D0F%2FC782%2FIMG-8242VOLL_5_ECK.JPG&shop=80300026
11,29 ct Crystal opal 600.00 € https://www.koroit-opal-company.com/storage/images/image?remote=https%3A%2F%2Fwww.koroit-opal-company.com%2FWebRoot%2FStore15%2FShops%2F80300026%2F6335%2FB93C%2F9276%2F3389%2FB532%2F0A0C%2F6D0B%2F3404%2FIMG-5671VOLL_5_ECK.JPG&shop=80300026

UPDATE: michalb93 asked how find api link:

1. Press "Show more" button with open dev tools on network tab
2. ctrl+f and search new opal's name, i search "2,75 ct Crystal opal"
3. After pressing Enter, u get list of requests which contain this name, u can see it in the attached screenshot.