how to load the full web page before start downloading with requests.get()?

Question

My program asks user for a keyword, downloads all the images from the https://www.pexels.com/ and stores them in a folder on the hard-drive.
The problem is, it downloads only the first 30 pictures that appears when the page loads, but doesn't take into consideration that when I scroll down, more images load in the page.

I want my program "scroll down" the page and download all images. Here is my code:

#! /usr/bin/python3 
import os, requests, bs4
keyword = input('Enter one-word search keyword: ')
url = 'https://www.pexels.com/search/' + keyword
res = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"})
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'html.parser')
tagObj = soup.select('.photo-item__img')

if tagObj == []:
    print('Sorry, no pictures found!')
else:
    print(len(tagObj))
    os.makedirs(str(keyword), exist_ok=True)
    for i in range(len(tagObj)):
        imgUrl = tagObj[i].get('srcset')
        print('Downloading img %s' %imgUrl)
        res = requests.get(imgUrl, headers={"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"})
        res.raise_for_status()
        # open img file for binary writing.
        imgFile = open(os.path.join(str(keyword), os.path.basename(imgUrl)), 'wb')
        for chunk in res.iter_content(100000):
            imgFile.write(chunk)
        imgFile.close()
    print('Done.')

Kamal · Accepted Answer

The webpage is loading only first 30 results and loads more with xhr request when scrolled. Using devtools of browser, I found the actual xhr request and then used it to get all the data.

Sample URL for XHR get request:

https://www.pexels.com/search/iphone%20x/?format=js&seed=2019-03-19%2B06%3A31%3A20%2B%2B0000&page=2

Sample response from the request:

;(function() {
  var infiniteScrollingAppender = window.Pexels.PhotoGrid.infiniteScrollingAppender({
    currentPage: 26,
    totalPages: 26,
    paginationHtml: 'Previous 1 2 … 18 19 20 21 22 23 24 25 26 Next',
    inlineSponsoredPhotosUrl: '/sponsored_photos/8/inline/?query=iphone+x'
  });

  infiniteScrollingAppender.append('








  





Nick Demou









', 0);    
  infiniteScrollingAppender.execute()
})();

You can use your own way to parse the response and find the required data. The following code will collect the same "srcset" data of all the images as done in your code using BeautifulSoup(Note: The full response can not be made into soup as it is not a valid HTML). You can merge your downloading code with it.

import datetime, requests
from bs4 import BeautifulSoup
seed = datetime.datetime.now().strftime('%Y-%m-%d%%2B%H%%3A%M%%3A%S%%2B%%2B0000')
url = 'https://www.pexels.com/search/{}/?format=js&seed={}&page='.format(keyword, seed)
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36'}
res = requests.get(url + '1', headers=headers)
# Extract total number of pages of results from response like "totalPages: 26,"
pages = int(res.text[res.text.find('totalPages')+11:res.text.find(',',res.text.find('totalPages')+11)])
imgurls = []

if not pages:
    print('Sorry, no pictures found!')
else:
    for page in range(1, pages+1):
        # Every new result of search is added after this string, so splitting response text with it.
        imgs = res.text.split('infiniteScrollingAppender.append')[1:]
        for img in imgs:
            # The response text has escaped single and double quotes with backslash, so replacing them to get valid html.
            soup = BeautifulSoup(img[2:-5].replace("\'", "'").replace('\"', '"'), 'html.parser')
            imgurls.append(soup.select('.photo-item__img')[0].get('srcset'))
        if page < pages:
            res = requests.get(url + str(page+1), headers=headers)

Let me know if you face any issue.

UPDATE:

You can find such xhr requests yourself in future using devtools of browser. For this case, open devtools in chrome and go to 'Network' tab, filter to show only XHR requests and then scroll to load more results. It will show a request like the above sample.

how to load the full web page before start downloading with requests.get()?

Answers (1)

Related Questions