CODEWITHSUNDEEP
pythonbeautifulsoup
Itay Tsuk
Itay Tsuk

Reputation: 103

BeautifulSoup.find_all doesn't find everything- Python

I used this code here to find image's URLs on a pictures website:

import requests
from bs4 import BeautifulSoup
import urllib.request

inp = input("search:")

url= "https://unsplash.com/s/photos/" + inp

response = requests.get(url)

soup = BeautifulSoup(response.content, "html.parser")

images = soup.find_all("img", attrs={"class": "_2UpQX"})
print(len(images)

Although it finds the first pictures on the page it doesn't find all of them or even close to this (like 6/500 or something like that)

What is wrong with this code? Is it because the page doesn't load all of itself?

Thank you for helping.

Upvotes: 1

Views: 69

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195408

The data is loaded dynamically via JavaScript in JSON format. To load the image URLs from various pages, you can use this example:

import json
import requests


search = "tree"

api_url = "https://unsplash.com/napi/search/photos"
query = {"query": search, "per_page": "20", "page": "1", "xp": ""}

for page in range(1, 3):  # <--- increase number of pages here
    query["page"] = page
    data = requests.get(api_url, params=query).json()
    for result in data["results"]:
        print(result["urls"]["full"])
    print("-" * 80)

Prints:

https://images.unsplash.com/photo-1556108936-019b2ccd9c11?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8MzgxfHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1602280493900-4570f57b9efc?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8MzgyfHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1614691211055-dee3dc6eb80d?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8MzgzfHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1593052583071-d5c8bb438720?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzg0fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1579964208918-0e6752566887?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzg1fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1586660491914-63f40d885b97?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzg2fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1588249888625-8ac37a9f264d?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzg3fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1615146064145-14d6e59650fc?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzg4fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1610193450857-1cb58cb64b8c?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzg5fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1587691812967-d0dc0b0684c6?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8MzkwfHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1607808137325-4e0c8b04b23b?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8MzkxfHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1602592818543-90a1b47dc967?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8MzkyfHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1591969739980-0d99dcaa71b3?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8MzkzfHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1597517339515-ec57f0de7561?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzk0fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1614616475159-9eb631ad0873?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzk1fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1609700931477-11b704af81c8?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzk2fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1611488533271-e953994e0877?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzk3fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1616636584787-207781fc78ef?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzk4fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1541500233866-71164d920e0b?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mzk5fHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1553537093-d3d970825195?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8NDAwfHx0cmVlfGVufDB8fHx8MTYyMTA5NDc1OQ&ixlib=rb-1.2.1&q=85
--------------------------------------------------------------------------------
https://images.unsplash.com/photo-1607670783743-4132d134b122?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8NzgxfHx0cmVlfGVufDB8fHx8MTYyMTA3MTcwNA&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1613477092384-6ada43052438?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8NzgyfHx0cmVlfGVufDB8fHx8MTYyMTA3MTcwNA&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1507144364888-ac3e6be5d3b4?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8NzgzfHx0cmVlfGVufDB8fHx8MTYyMTA3MTcwNA&ixlib=rb-1.2.1&q=85
https://images.unsplash.com/photo-1577451462333-e969f7cf989c?crop=entropy&cs=srgb&fm=jpg&ixid=MnwxMjA3fDB8MXxzZWFyY2h8Nzg0fHx0cmVlfGVufDB8fHx8MTYyMTA3MTcwNA&ixlib=rb-1.2.1&q=85

...and so on.

Upvotes: 2

Related Questions