CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Mike
Mike

Reputation: 161

Web Scraping on website with changing codes for every <div>

I'm doing some webscraping on this website and I'm facing a problem in the soup.select function I'm trying to use. Basically this is their HTML code:

<li class="Features_nd-list__item__3hWVx Features_in-feat__item__2-hIE Features_in-feat__item--main__3EFFl RealEstateListCard_in-realEstateListCard__features--main__2uSci">€ 489.000</li>

I need to select this list item to get the price listed for this ad. Problem is that they recently changed the HTMl code and included codes that change randomly for every element in their HTML code, problably to block what I'm trying to do...

Example:

<li class="Features_nd-list__item__3hWVx

This 3hWVx is the part that is everchanging and having to select every price in every add with changing codes automatically, I can't find away to select it. Here is the solution I found for href, but I can't solve it for other parts of code.

My code:

        soup = BeautifulSoup(requests.get(userurl).content, "html.parser")
    for link in soup.select('a[href^="https://www.immobiliare.it/annunci/"]'):
        list_links.append(link["href"])
        list_titles.append(link.get_text(strip=True))
    for link in soup.select('li[class^="Features_nd-list__item__"]'):
        if "€" in link:
            list_prices.append(link.get_text(strip=True))

I hope you guys/gals can help me out as it's really difficult to understand and solve this problem.

Thanks!

Upvotes: 2

Views: 622

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195553

Try:

import requests
from bs4 import BeautifulSoup

# for example this url:
url = "https://www.immobiliare.it/vendita-case/agrigento/"

soup = BeautifulSoup(requests.get(url).content, "html.parser")
list_links = []
list_titles = []
list_prices = []

for ad in soup.select('li[id^="link_ad"]'):
    list_links.append(ad.a["href"])
    list_titles.append(ad.a.get_text(strip=True))
    price = ad.find(lambda t: t.name == "li" and "€" in t.text)
    # alternative:
    # price = ad.select_one('li:-soup-contains("€")')
    list_prices.append(price.text if price else "-")

# pretty print the lists:
for t, p, l in zip(list_titles, list_prices, list_links):
    print("{:<50}  {:<10}  {:<40}".format(t[:50], p, l))

Prints:

Appartamento via Parco Angeli, San Leone, Villaset  € 165.000   https://www.immobiliare.it/annunci/89607425/
Trilocale Cortile Bosio, Centro Storico, Agrigento  € 139.000   https://www.immobiliare.it/annunci/89607405/
Appartamento via Piersanti Mattarella, Via Imera,   € 129.000   https://www.immobiliare.it/annunci/89607367/
Quadrilocale via Empedocle, Centro Storico, Agrige  € 127.000   https://www.immobiliare.it/annunci/89607385/
Bilocale via Atenea, Centro Storico, Agrigento      € 57.000    https://www.immobiliare.it/annunci/89607351/
Quadrilocale piazza Fratelli Rosselli, Via Imera,   -           https://www.immobiliare.it/annunci/89607329/
Quadrilocale viale della Vittoria, Via Imera, V. d  -           https://www.immobiliare.it/annunci/89607437/
Villa unifamiliare viale dei Giardini 67, San Leon  € 680.000   https://www.immobiliare.it/annunci/89595189/
Appartamento via Parco Angeli, San Leone, Villaset  € 165.000   https://www.immobiliare.it/annunci/89566323/
Trilocale Cortile Bosio, Centro Storico, Agrigento  € 139.000   https://www.immobiliare.it/annunci/89566319/
Appartamento via Piersanti Mattarella, Via Imera,   € 129.000   https://www.immobiliare.it/annunci/89566261/
Trilocale via dei Borboni, San Leone, Villaseta, V  € 57.000    https://www.immobiliare.it/annunci/89566253/
Quadrilocale via Empedocle, Centro Storico, Agrige  € 127.000   https://www.immobiliare.it/annunci/89566313/
Bilocale via Michele Lizzi, San Leone, Villaseta,   € 99.000    https://www.immobiliare.it/annunci/89566247/
Bilocale via Atenea, Centro Storico, Agrigento      € 57.000    https://www.immobiliare.it/annunci/89566263/
Quadrilocale piazza Fratelli Rosselli, Via Imera,   -           https://www.immobiliare.it/annunci/89566257/
Appartamento viale della Vittoria, Via Imera, V. d  -           https://www.immobiliare.it/annunci/89566329/
Villa unifamiliare via Vittorio De Sica, San Leone  € 169.000   https://www.immobiliare.it/annunci/89566171/
Trilocale via Michele Amari, San Leone, Villaseta,  € 48.000    https://www.immobiliare.it/annunci/89574519/
Appartamento via Parco Angeli, San Leone, Villaset  € 165.000   https://www.immobiliare.it/annunci/89550179/
Trilocale Cortile Bosio, Centro Storico, Agrigento  € 139.000   https://www.immobiliare.it/annunci/89550131/
Appartamento via Piersanti Mattarella, Via Imera,   € 129.000   https://www.immobiliare.it/annunci/89549999/
Quadrilocale via Empedocle, Centro Storico, Agrige  € 127.000   https://www.immobiliare.it/annunci/89549901/
Appartamento piazza Fontanelle 28, Serraferlicchio  € 78.000    https://www.immobiliare.it/annunci/89524063/
Appartamento viale della Vittoria, Via Imera, V. d  -           https://www.immobiliare.it/annunci/89550241/

Upvotes: 1

Related Questions