CODEWITHSUNDEEP
pythonpython-3.xloopsweb-scrapingbeautifulsoup
Craigcellent
Craigcellent

Reputation: 3

New to webscraper and looping data unexpectedly

I'm fresh off some intro courses to programming, mostly Python, and I'm trying to write a basic scraper to extract data on used Tesla's from a specific site (later to add diff sites and store in a DB and do some data analysis etc with SQL) and I'm running into issues right out of the gate.

I'm trying to pull 8 pieces of information, the last 3 within the same div, rather than looping the information (Listing Date, Location and Miles) for each car, it's spitting that information back for ALL cars in every loop.

I know the problem is somewhere in my looping code starting "for listing_details" but any time I try to adjust different elements I blow it up with a traceback error.

Here's the code below, I sincerely hope someone may be able to point out my obvious oversight!

from bs4 import BeautifulSoup
import requests

source = requests.get('https://onlyusedtesla.com/listings/?_sft_model=model-s&_sfm_out_asking_price=699+150004&_sfm_state=Arizona').text

soup = BeautifulSoup(source, 'lxml')

for listing in soup.find_all('div', class_='out-list-item'):

    listing_info = listing.h2.a.text
    listing_separate = listing_info.split("/")

    for list in listing_separate:
        print(list)


    price_info = listing.find('div', class_='asking-price')
    price = price_info.text
    p = price.split(' ')
    print(p[3], p[10])


    for listing_details in soup.find_all('div', class_='out-summary-data small-12 medium-4 columns'):

        details = listing_details
        listing_date = details.find_all("p")[0]
        ld = listing_date.text
        print(ld)
        location = details.find_all("p")[1]
        loc = location.text
        print(loc)
        mileage = details.find_all("p")[3]
        mil = mileage.text
        print(mil)

Upvotes: 0

Views: 59

Answers (1)

αԋɱҽԃ αмєяιcαη
αԋɱҽԃ αмєяιcαη

Reputation: 11505

import requests
from bs4 import BeautifulSoup

r = requests.get(
    'https://onlyusedtesla.com/listings/?_sft_model=model-s&_sfm_out_asking_price=699+150004&_sfm_state=Arizona')
soup = BeautifulSoup(r.text, 'html.parser')

h2 = []
asking = []
details = []
for item in soup.findAll('div', attrs={'class': 'small-12 large-8 columns'}):
    for name in item.findAll('h2'):
        h2.append(name.text)
for item in soup.findAll('div', attrs={'class': 'asking-price'}):
    item = item.text.replace("       ", "")
    asking.append(item[15:])
for item in soup.findAll('div', attrs={'class': 'out-summary-data small-12 medium-4 columns'}):
    item = [name.get_text(strip=True) for name in item.findAll('p')]
    details.append(item)

for a, b, c in zip(h2, asking, details):
    print(a, b, *c)

Output:

  2015 / Model S / P90DL / Dark Blue  55,000.00 Nov 21, 2019 Gilbert, Arizona Excellent 76,000 miles
  2015 / Model S / 85 / White  39,950.00 Nov 20, 2019 GOODYEAR, Arizona Excellent 100,500 miles
  2017 / Model S / 75 / Pearl White  63,000.00 Nov 11, 2019 Tempe, Arizona Excellent 20,436 miles
  2016 / Model S / 75 / Midnight Silver Metallic  48,000.00 Oct 6, 2019 Phoenix, Arizona Excellent 27,703 miles
  2017 / Model S / 90D / Metallic Blue  62,950.00 Oct 2, 2019 Chandler, Arizona Excellent 48,000 miles
  2017 / Model S / 75D / Midnight Silver  57,000.00 Aug 10, 2019 Phoenix, Arizona Excellent 26,500 miles
  2014 / Model S / 85 / Grey Metallic  39,950.00 Aug 3, 2019 Gilbert, Arizona Very Good 68,861 miles
  2017 / Model S / 100D / Deep Blue Metallic  72,500.00 Jul 1, 2019 Phoenix, Arizona Excellent 17,000 miles
  2016 / Model S / 75D / Midnight Silver  49,495.00 Feb 24, 2019 Gilbert, Arizona Excellent 20,157 miles
  2012 / Model S / P85 / Black  32,000.00 Feb 19, 2019 Mesa, Arizona Very Good 133,000 miles
  2018 / Model S / 75D / Red  74,950.00 Feb 5, 2019 Tucson, Arizona Excellent 8,964 miles
  2014 / Model S / P85D / Midnight Silver  70,950.00 Sep 13, 2018 Phoenix, Arizona Excellent 13,800 miles
  2013 / Model S / 60 / Red  37,500.00 Jun 27, 2018 Chandler, Arizona Good 73,536 miles
  2018 / Model S / 75D / Blue. EAP  72,950.00 Jun 20, 2018 Queen Creek, Arizona Excellent 2,279 miles
  2014 / Model S / P85+ / Black  69,900.00 Jun 18, 2018 Gilbert, Arizona Excellent 48,500 miles
  2014 / Model S / 85 / Blue  44,950.00 May 3, 2018 Tempe, Arizona Excellent 68,000 miles
  2013 / Model S / 85 / Silver  41,000.00 Apr 18, 2018 Chandler, Arizona Excellent 59,000 miles
  2013 / Model S / 85 / White  49,850.00 Apr 10, 2018 Queen Creek, Arizona Excellent 39,937 miles
  2014 / Model S / 85 / Dark Blue. Autopilot  57,000.00 Mar 28, 2018 Mesa, Arizona Excellent 59,077 miles
  2014 / Model S / P85+ / Blue  58,000.00 Mar 18, 2018 Phoenix, Arizona Excellent 51,309 miles

Upvotes: 2

Related Questions