CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Sean Kelly
Sean Kelly

Reputation: 183

Simple Python Web-scraper with Beautiful Soup

I am trying to scrape car data. Their 'id' tags increment by 1, however i just can't seem to figure out how to do that. Here is what I have: 

import bs4 as bs
import urllib

source = urllib.request.urlopen('http://www.25thstauto.com/inventory.aspx?cursort=asc&pagesize=500').read()
soup = bs.BeautifulSoup(source, 'lxml')

#finds the total number of cars
count = soup.find('span', {'id': 'ctl00_cphBody_inv1_lblVehicleCount'}).getText()[:2]
count = int(count)

i = 1
for url in range(1,count):
url = soup.find_all('a', {'id': 'ctl00_cphBody_inv1_rptInventoryNew_ctl0'+i+'_nlVehicleDetailsTitle'})
    print(url['href'])
    i = i + 1

Upvotes: 1

Views: 223

Answers (1)

宏杰李
宏杰李

Reputation: 12178

import bs4 as bs
import urllib
import re

source = urllib.request.urlopen('http://www.25thstauto.com/inventory.aspx?cursort=asc&pagesize=500').read()
soup = bs.BeautifulSoup(source, 'lxml')

for a in soup.find_all('a', id=re.compile('ctl00_cphBody_inv1_rptInventoryNew')):
    print(a.get('href'))

out:

2008_Chevrolet_Malibu_Easton_PA_265928462.veh
2008_Chevrolet_Malibu_Easton_PA_265928462.veh
2008_Chevrolet_Malibu_Easton_PA_265928462.veh
2002_Nissan_Xterra_Easton_PA_266894015.veh
2002_Nissan_Xterra_Easton_PA_266894015.veh
2002_Nissan_Xterra_Easton_PA_266894015.veh
2009_Chevrolet_Cobalt_Easton_PA_265621796.veh
2009_Chevrolet_Cobalt_Easton_PA_265621796.veh

Use regex to find a tag whose id attribute contain ctl00_cphBody_inv1_rptInventoryNew

OR use CSS selector:

for a in soup.select('a[id*=ctl00_cphBody_inv1_rptInventoryNew]'):
    print(a.get('href'))

The idea is same.

Upvotes: 1

Related Questions