CODEWITHSUNDEEP
pythonbeautifulsouppython-requests
hello543
hello543

Reputation: 137

Python BeautifulSoup/Requests Loop through pages - doesn't work

I have tried suggestions from the forum to loop through multiple web pages, but it does not work:

for i in range(1,5):
    url = f'https://www.xxxx.com/?page={i}' # just a random web page
    result = requests.get(url).text
    doc = BeautifulSoup(result, 'html.parser')

it gives only the first page result so https://www.xxxx.com/?page=1

any suggestion what is wrong here?

Upvotes: 0

Views: 652

Answers (1)

codeBB
codeBB

Reputation: 83

it would be very helpful if you provide an Error StackTrace. In general, it is possible to loop scrape multiple pages at once.

Here is a code that worked for me with an arbitrary web page.

#!pip install requests
#!pip install beautifulsoup4

import requests
from bs4 import BeautifulSoup as bs
page = 1
titles = []
for i in range(1,5):
      url = f"https://www.bookdepository.com/bestsellers?page={page}"
      response = requests.get(url)
      html = response.content
      soup = bs(html, "lxml")
      for h3 in soup.find_all("h3", class_="title"):
            titles.append(h3.get_text(strip=True))
      page = page + 1

print(titles)

A simple google search revelead the code. https://data36.com/scrape-multiple-web-pages-beautiful-soup-tutorial/

The range was modified as shown in your example.

Upvotes: 1

Related Questions