CODEWITHSUNDEEP
pythonloopsweb-scrapingbeautifulsoupelement
poldy
poldy

Reputation: 204

Iterating through page elements beautifulsoup

I'm trying to build a quick scraper for a popular car site. I can get back results for one car but I cannot figure out how to return all cars on the page. findAll() is throwing an error. Any help would be appreciated

from bs4 import BeautifulSoup
import requests

#search = input('Enter car to search: ')
url = 'https://www.donedeal.ie/cars?words=bmw' #+ search
site = requests.get(url)
page = site.content
soup = BeautifulSoup(page, 'html.parser')
print("URL: ", site.url)

if site.status_code == 200:
    print("HTTP Status: ", site.status_code, "\n")
else:
    print("Bad HTTP response", "\n")

cars = soup.find('div', attrs={'class': 'top-info'})
county = soup.find('span', attrs={'class': 'county-disp icon-pin'})
span = cars.find('span')

for result in span:
    for result2 in county:
        print(result, "-", result2)

Upvotes: 2

Views: 753

Answers (1)

Aziz Alto
Aziz Alto

Reputation: 20311

I am not sure which information you want to extract. Assuming you want the car type and the county info, findAll() works with something like this:

>>> cars = soup.findAll('div', attrs={'class': 'top-info'})
>>> for car in cars:
...     loc = car.find('span', attrs={'class': 'county-disp icon-pin'})
...     if loc:
...         print('type:', car.text, 'location:', loc.text)
...     else:
...         print('type:', car.text)
type: Bmw 320 CdTipperary location: Tipperary
type: Bmw 520d MsportDonegal location: Donegal
type: BMW2004
type: BMW2010
type: Bmw2010
type: Bmw2000
type: Bmw2001
type: Bmw2004
type: Bmw2004
type: bmw2003
type: BMW2009
type: Bmw2010
type: Bmw1990
type: BMW2004
type: BMW2012
type: Bmw2000
type: bmw2001
type: BMW2004
type: BMW2008
type: BMW2005
type: Bmw2006
type: Bmw2002
type: BMW2004
type: Bmw2000
type: BMW2003
type: BMW2011
type: BMW2001
type: Bmw2000
type: Bmw2002
type: BMW2007

note that for one page only. You will have to do the other pages urls.

Upvotes: 2

Related Questions