CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
sgl
sgl

Reputation: 95

Beautiful soup does not go to the next page

I am trying to navigate to the second page of the URL to get the street names, but the results are still from the first page.

When I surf manually to this page it gives me the second page of the url, but when using the code it goes still to the first page.

My code:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://miqa.nl/woningen/koop/#page/2;')
soup = BeautifulSoup(page.text, 'html.parser')

title_div = soup.find_all('div', class_='title')
for streets in title_div :
     street = streets.find('h2').text
     print(street)

Does someone know why this happens?

Upvotes: 0

Views: 171

Answers (2)

Samsul Islam
Samsul Islam

Reputation: 2609

I was trying to solve the next page navigation extracting data. Here is my code.

def miqa(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    title_div = soup.find_all('div', class_='title')
    for streets in title_div :
        street = streets.find('h2').text
        print(street)
    try:
        page = soup.select_one('a.next.page-numbers', href=True).get('href')
        print('URL', page)
        miqa(page)
    except AttributeError:
        print('There is no page to extract data. Finished.')


if __name__ == "__main__":
    url = 'https://miqa.nl/woningen/koop/#q1YqqSxIVbJSys7PL1DSUcovSkktSqoECqQklpTmeuaV5acWWaWkFicr1QIA'
    print('starting ...')
    print(url)
    miqa(url)

Upvotes: 0

dimay
dimay

Reputation: 2804

Try this:

import requests
from bs4 import BeautifulSoup

for page in range(1,4):
    print("------", page, "---------")
    r = requests.get(f'https://miqa.nl/woningen/koop/page/{page}/')
    soup = BeautifulSoup(r.text, 'html.parser')

    title_div = soup.find_all('div', class_='title')
    for streets in title_div :
        street = streets.find('h2').text
        print(street)

Upvotes: 1

Related Questions