CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Sarah
Sarah

Reputation: 617

How to get all data using beautifulsoup?

I am trying to scrape all the addresses for "Recent Sales" in this page: https://www.compass.com/agents/irene-vuong/

My current code looks like:

url = 'https://www.compass.com/agents/irene-vuong/'
url = requests.get(url)
soup = BeautifulSoup(url.text, 'html')

for item in soup.findAll('div', attrs={'class': 'uc-listingCard-content'}):
    new = item.find('a', attrs={'class': 'uc-listingCard-title'})
    print(new.text)

My output is :

256-258 Wyckoff Street
1320 Glenwood Road
1473 East 55th Street
145 Winter Avenue
25-02 Brookhaven Avenue

which is the addresses of "current" listings.
My expected output is:

352 94th Street
1754 West 12th Street
2283 E 23rd st
2063 Brown Street
3423 Avenue U
2256 Stuart Street

Which are the addresses under "Recent Sales". No matter what, I only get current listing addresses, but not all listing addresses. I tried to use re.compile(r'Recent Sales') but it would not work. I'm not sure how to get to "Recent Sales".

Any help will be greatly appreciated.

+++++ I also tried to use text 'Recent Sales' as below:

for item in soup.findAll(text=re.compile(r'Recent Sales')).findNext():
    for i in item.find('div', attrs={'class':'profile-acive-listings'}):
        new = i.find('a', attrs={'class': 'uc-listingCard-title'})
        print(new.text)

But I get an error of:

AttributeError: ResultSet object has no attribute 'findNext'. You're probably treating a list of items like a single item. Did you call find_all() when you meant to call find()?

+++ Also tried to use class data-tn : recent sales:

for item in soup.findAll('div', attrs={'data-tn':'recent-sales'}):
    new = item.findAll('a', attrs={'class': 'uc-listingCard-title'})
    print(new.text)

But it won't return anything.

Upvotes: 1

Views: 617

Answers (2)

petezurich
petezurich

Reputation: 10194

You can use Selenium. It renders your page in an automated browser. From the rendered page you can then get the full HTML and retrieve your listings.

Try this:

from selenium import webdriver
from bs4 import BeautifulSoup

browser = webdriver.Firefox()
browser.get("https://www.compass.com/agents/irene-vuong/")
html = browser.page_source
soup = BeautifulSoup(html, 'html')

for item in soup.findAll('div', attrs={'class': 'uc-listingCard-content'}):
    new = item.find('a', attrs={'class': 'uc-listingCard-title'})
    print(new.text)

This prints out:

256-258 Wyckoff Street
1320 Glenwood Road
1473 East 55th Street
145 Winter Avenue
25-02 Brookhaven Avenue
352 94th Street
1754 West 12th Street
2283 E 23rd St
2063 Brown Street
3423 Avenue U
2256 Stuart Street
East 61st Street

Edit:

If you want to parse the data from the raw HTML you have to get it a script tag.

Try this:

import json
from bs4 import BeautifulSoup
import requests
import pandas as pd

url = 'https://www.compass.com/agents/irene-vuong/'
res = requests.get(url)
soup = BeautifulSoup(res.content, 'html')

script = soup.find_all("script")[4]
data = json.loads(script.text.split("window.__AGENT_PROFILE__ = ")[1])
data = data["data"]

df_sales = pd.DataFrame(data["closedDeals"]["sales"])
df_rentals = pd.DataFrame(data["closedDeals"]["rentals"])

This gives you Pandas dataframes with all the listing data like this.

listingIdSHA    listingType     location    size    price   detailedInfo    media   dealInfo    isOffMLS    pageLink    pageLinkSlug    canonicalPageLink   userListingCompliance
0   210837948508195937  2   {'prettyAddress': '352 94th Street', 'city': '...   {'bedrooms': 4, 'bathrooms': 2.75}  {'lastKnown': 1250000, 'formatted': '$1,250,000'}   {'amenities': ['Driveway', 'Open Kitchen', 'Ga...   [{'category': 0, 'thumbnailUrl': 'https://d278...   {'disclaimer': 'No guarantee, warranty or repr...   False   /listing/352-94th-street-brooklyn-ny-11209/210...   352-94th-street-brooklyn-ny-11209   /listing/352-94th-street-brooklyn-ny-11209/210...   {'descriptionCompliance': 0}
1   122690464561282785  2   {'prettyAddress': '1754 West 12th Street', 'ci...   {'bedrooms': 4, 'bathrooms': 2}     {'lastKnown': 1040000, 'formatted': '$1,040,000'}   {'amenities': ['Basement', 'Private Outdoor Sp...   [{'category': 0, 'thumbnailUrl': 'https://d278...   {'disclaimer': 'No guarantee, warranty or repr...   False   /listing/1754-west-12th-street-brooklyn-ny-112...   1754-west-12th-street-brooklyn-ny-11223     /listing/1754-west-12th-street-brooklyn-ny-112...   {'descriptionCompliance': 0}
2   NaN     2   {'prettyAddress': '2283 E 23rd St', 'neighborh...   {'bedrooms': 3, 'bathrooms': 2}     {'lastKnown': 800000, 'formatted': '$800,000'}  NaN     [{'category': 0, 'thumbnailUrl': 'https://d278...   {'disclaimer': 'No guarantee, warranty or repr...   False   NaN     2283-e-23rd-st  NaN     NaN
3   235974146369023201  2   {'prettyAddress': '2063 Brown Street', 'city':...   {'bedrooms': 3, 'bathrooms': 2}     {'lastKnown': 755000, 'formatted': '$755,000'}  NaN     [{'category': 0, 'thumbnailUrl': 'https://d278...   {'disclaimer': 'No guarantee, warranty or repr...   False   /listing/2063-brown-street-brooklyn-ny-11229/2...   2063-brown-street-brooklyn-ny-11229     /listing/2063-brown-street-brooklyn-ny-11229/2...   {'descriptionCompliance': 0}
4   186865317970981409  2   {'prettyAddress': '3423 Avenue U', 'city': 'Br...   {'bedrooms': 5, 'bathrooms': 2}     {'lastKnown': 627000, 'formatted': '$627,000'}  {'amenities': ['Hardwood Floors', 'Garage', 'C...   [{'category': 0, 'thumbnailUrl': 'https://d278...   {'disclaimer': 'No guarantee, warranty or repr...   False   /listing/3423-avenue-u-brooklyn-ny-11234/18686...   3423-avenue-u-brooklyn-ny-11234     /listing/3423-avenue-u-brooklyn-ny-11234/18686...   {'descriptionCompliance': 0}
5   286987776170131617  2   {'prettyAddress': '2256 Stuart Street', 'city'...   {'bedrooms': 3, 'bathrooms': 1}     {'lastKnown': 533000, 'formatted': '$533,000'}  NaN     [{'category': 0, 'thumbnailUrl': 'https://d278...   {'disclaimer': 'No guarantee, warranty or repr...   False   /listing/2256-stuart-street-brooklyn-ny-11229/...   2256-stuart-street-brooklyn-ny-11229    /listing/2256-stuart-street-brooklyn-ny-11229/...

To retrieve just the listing adresses use this further step:

from pandas import json_normalize

df_sales = df_sales.location.apply(lambda x: dict(x))
df_sales = json_normalize(df_sales)

df_rentals = df_rentals.location.apply(lambda x: dict(x))
df_rentals = json_normalize(df_rentals)

Output:

prettyAddress   city    state   zipCode     geoId   neighborhood    subNeighborhoods
0   352 94th Street     Brooklyn    NY  11209   nyc     NaN     NaN
1   1754 West 12th Street   Brooklyn    NY  11223   nyc     NaN     NaN
2   2283 E 23rd St  NaN     NaN     NaN     nyc     Sheepshead Bay  [Sheepshead Bay]
3   2063 Brown Street   Brooklyn    NY  11229   nyc     NaN     NaN
4   3423 Avenue U   Brooklyn    NY  11234   nyc     NaN     NaN
5   2256 Stuart Street  Brooklyn    NY  11229   nyc     NaN     NaN

Edit:

You can get more clean data like so:

df_sales = pd.DataFrame(data["closedDeals"]["sales"])

columns = ['listingIdSHA', 'listingType', 'location', 'size', 'price']
df_sales = df_sales[columns]

expanded_data = []

for column in ['location', 'size', 'price']:
    expanded = df_sales[column].apply(lambda x: dict(x))
    expanded_data.append(json_normalize(expanded))

expanded_data = pd.concat(expanded_data, axis=1)

df_sales_cleaned = pd.concat([df_sales[['listingIdSHA', 'listingType']], expanded_data], axis=1)
display(df_sales_cleaned)

Output:

listingIdSHA    listingType     prettyAddress   city    state   zipCode     geoId   neighborhood    subNeighborhoods    bedrooms    bathrooms   lastKnown   formatted
0   210837948508195937  2   352 94th Street     Brooklyn    NY  11209   nyc     NaN     NaN     4   2.75    1250000     $1,250,000
1   122690464561282785  2   1754 West 12th Street   Brooklyn    NY  11223   nyc     NaN     NaN     4   2.00    1040000     $1,040,000
2   NaN     2   2283 E 23rd St  NaN     NaN     NaN     nyc     Sheepshead Bay  [Sheepshead Bay]    3   2.00    800000  $800,000
3   235974146369023201  2   2063 Brown Street   Brooklyn    NY  11229   nyc     NaN     NaN     3   2.00    755000  $755,000
4   186865317970981409  2   3423 Avenue U   Brooklyn    NY  11234   nyc     NaN     NaN     5   2.00    627000  $627,000
5   286987776170131617  2   2256 Stuart Street  Brooklyn    NY  11229   nyc     NaN     NaN     3   1.00    533000  $533,000

Upvotes: 1

Izzey
Izzey

Reputation: 1

i recently got a project where I'm using this too, but even without Regex try the code like this

for item in soup.findAll(text=re.compile(r'Recent Sales')):
    for i in item.encode_contents().find('div', {'class':'profile-acive-listings'}):
        new = i.find('a', {'class': 'uc-listingCard-title'})
        print(new.text)

Upvotes: 0

Related Questions