CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Spirula
Spirula

Reputation: 57

webscraper no longer retrieving data - can still access website via browser

I'm new to webscraping and have been trying for fun to scrape a boxing website. My code below was working on the first attempt, and when I tried to re-run it, it was no longer retrieving the link data any more. I can still access the website from my browser, so not sure what the error is! Appreciate any pointers.

import os
from urllib.request import urlopen, Request
from bs4 import BeautifulSoup
import re

os.system('cls')

heavy = 'https://boxrec.com/en/ratings?r%5Brole%5D=box-pro&r%5Bsex%5D=M&r%5Bstatus%5D=a&r%5Bdivision%5D=Heavyweight&r%5Bcountry%5D=&r_go='

pages = set()

def get_links(page_url):
    print("running crawler...")
    global pages
    req = Request(heavy, headers = {'User-Agent':'Mozilla/5.0'})
    html = urlopen(req)
    bs = BeautifulSoup(html.read(), 'html.parser')
    for link in bs.find_all('a', href=re.compile('^(/en/box-pro/)')):
        if 'href' in link.attrs:
            if link.attrs['href'] not in pages:
                new_page = link.attrs['href']
                print(new_page)
                pages.add(new_page)
                get_links(new_page)

get_links('')

print("crawling done.")

Upvotes: 0

Views: 101

Answers (0)

Related Questions