CODEWITHSUNDEEP
web-scrapingbeautifulsouppython-requests
Sayeed
Sayeed

Reputation: 29

Web Scraping: Some Pages URL are missing

I was trying to scrap the first 9 pages of a website but it looks like page5 and page7 are missing. This is making show python an attribute error. However, I think an 'if' function can solve this but I'm unable to figure out the code for the if function. Here is my code

import requests
from bs4 import BeautifulSoup
base_url="http://cbcs.fastvturesults.com/student/1sp15me00"
for page in range(1,10,1):
    r=requests.get(base_url+str(page))
    c=r.content
    soup=BeautifulSoup(c,"html.parser")
    items=soup.find(class_="text-muted")
    if ??????????:
        pass
    else:
        print("{}\n{}".format(items.previous_sibling,items.text))

Upvotes: 0

Views: 216

Answers (2)

SIM
SIM

Reputation: 22440

You don't need to create else block here. Only checking if items is not None suffices. Try the below approach:

items = soup.find(class_="text-muted")
if items:
    print("{}\n{}".format(items.previous_sibling,items.text))

Upvotes: 1

nmog
nmog

Reputation: 236

The error occurs when you try to access attributes of items when items is set to None. This is done when BeautifulSoup cannot find anything with class_="text-muted"

The solution:

if not items:
    continue

Note that pass(from your solution) will just pass the current statement and move on to the next line in the loop. continue will end the current iteration and move on to the next iteration.

Upvotes: 2

Related Questions