CODEWITHSUNDEEP
pythonweb-scraping
Aedam
Aedam

Reputation: 141

how to add a loop to Python script that scrapes a website

I have a script that scrapes a website. However, I am looking for it to incrementally scrape the websites for a range. So imagine the range is set to 0-999. The code is:

import requests
from bs4 import BeautifulSoup

URL = 'https://www.greekrank.com/uni/1/sororities/'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')

uni = soup.find_all('h1', class_='overviewhead')
for title in uni:
    print(title.text)

rows = soup.find_all('div', class_='desktop-view')
for row in rows:
    print(row.text)

It would go to https://www.greekrank.com/uni/1/sororities/ scrape that, then go to https://www.greekrank.com/uni/2/sororities/ scrape that, etc.

Upvotes: 1

Views: 92

Answers (1)

Badgy
Badgy

Reputation: 819

Wrap it all in a loop. Also note the URL assignment.

import requests
from bs4 import BeautifulSoup

for x in range(0, 999):
    URL = f'https://www.greekrank.com/uni/{x}/sororities/'
    page = requests.get(URL)

    soup = BeautifulSoup(page.content, 'html.parser')

    uni = soup.find_all('h1', class_='overviewhead')
    for title in uni:
        print(title.text)

    rows = soup.find_all('div', class_='desktop-view')
    for row in rows:
        print(row.text)

Upvotes: 3

Related Questions