CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoupaliexpress
Borealis
Borealis

Reputation: 13

Aliexpress web scraper not repeating

I'm a complete beginner but have a question about my code for web scraping products on aliexpress.

The problem is that I only get 1 result instead of all of them.

from bs4 import BeautifulSoup
import requests

html_text = requests.get('https://dutch.alibaba.com/products/uhf_rfid_label.html?IndexArea=product_en&page=1').text
soup = BeautifulSoup(html_text, 'lxml')
producten = soup.find_all('div', class_ ='organic-list app-organic-search__list')
for product in producten:
    product_naam = product.find('p', class_ = 'elements-title-normal__content large').text
    jaren_actief = product.find('span', class_ = 'seller-tag__year flex-no-shrink').text
    print(f'''Product naam: {product_naam} ''')

How do I get the information of ALL the products?

Upvotes: 0

Views: 400

Answers (2)

ezzeddin
ezzeddin

Reputation: 521

Will assume the information you want to get is the title of the product and the price as @Andrej_Kesley suggested in his answer.

Parsing the page as HTML, you can get only 8 products with Beautiful Soup as follows:

from bs4 import BeautifulSoup
import requests

url = 'https://dutch.alibaba.com/products/uhf_rfid_label.html?IndexArea=product_en&page=1'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
products = soup.find_all('a', class_='elements-title-normal')
prices = soup.find_all('span', class_='elements-offer-price-normal__price')
num_products = len(products)

for i in range(num_products):
    print("{:<20} {}".format(prices[i].text, products[i].text))

Output:

US$ 0,02-US$ 0,10    Aangepaste Tag Sticker Labels Lange Bereik Goedkope Passieve Papier Roll Uhf Rfid Label
US$ 0,08-US$ 0,12    Magazijn & Asset & Productie Lijnmanagement Lange Afstand Alien H3 Chip Uhf Rfid Papier Label
US$ 0,78-US$ 0,85    Hopeland hot selling UHF RFID Animal Ear Tag mini uhf rfid ear tag 860 960MHz 5m reading range rfid label tag uhf
US$ 0,03-US$ 0,06    Factory Outlet Uhf Rfid Sticker/Label Met Chip
US$ 0,07-US$ 0,12    Long Range Goedkope Passieve Papier Roll Uhf Rfid Chip Label Tag Sticker
US$ 0,02-US$ 0,04    Groothandel Asset Tracking R6 Chip Uhf Papier Label Rfid Uhf Inventaris Labels Uhf Sticker
US$ 0,04-US$ 0,13    Aangepaste Tags Alien H3 9662 H9 9640/M4E Chip Long Range Passieve Uhf Rfid Tag/ Label/ Sticker
US$ 0,08-US$ 0,20    Full Color Afdrukken Hf/Uhf Passieve Papier Roll Smart Nfc Rfid Label/Sticker/Tag

Upvotes: 0

Andrej Kesely
Andrej Kesely

Reputation: 195573

The information is about products (price, title, etc.) is embedded within the HTML page in Javascript (in actual HTML are rendered only 8 products). You can use re/json module to parse it. For example:

import re
import json
import requests

html_text = requests.get(
    "https://dutch.alibaba.com/products/uhf_rfid_label.html?IndexArea=product_en&page=1"
).text

data = re.search(r"window\.__page__data__config = (\{.*\})", html_text).group(1)
data = json.loads(data)
# uncomment to print all data:
# print(json.dumps(data, indent=4))

for offer in data["props"]["offerResultData"]["offerList"]:
    print(
        "{:<20} {}".format(
            offer["tradePrice"]["price"], offer["information"]["puretitle"]
        )
    )

Prints:

US $0.02-$0.10       Aangepaste Tag Sticker Labels Lange Bereik Goedkope Passieve Papier Roll Uhf Rfid Label
US $0.07-$0.12       Long Range Goedkope Passieve Papier Roll Uhf Rfid Chip Label Tag Sticker
US $558.00-$688.90   Hopeland 15 meter uhf rfid scanner r2000 rfid reader device uhf scanner handheld terminal multi tag uhf rfid scanner
US $0.03-$0.06       Factory Outlet Uhf Rfid Sticker/Label Met Chip
US $558.00-$688.90   Hopeland Draagbare Uhf Rfid Terminal ISO18000 6C Multi-Tag Management Uhf Rfid Handheld Reader 2D Barcode Uhf Rfid Terminal
US $0.03-$0.12       Gratis Monster Waterdichte Nfc 213 Long Range Passieve Uhf Rfid Tag/ Label/ Sticker
US $0.06             Printable Uhf Rfid Adhesive Label/Rfid Sticker Tag/Rfid Tag Voor Boeken
US $0.04-$0.13       Aangepaste Tags Alien H3 9662 H9 9640/M4E Chip Long Range Passieve Uhf Rfid Tag/ Label/ Sticker
US $0.04-$0.12       Gratis Sample Lange Range Passieve Uhf Rfid Tag/ Label/ Sticker
US $0.03-$0.06       Aangepaste Tags Long Range Uhf Rfid Inlay/Natte Inlay/Label/Sticker
US $0.08-$0.20       Full Color Afdrukken Hf/Uhf Passieve Papier Roll Smart Nfc Rfid Label/Sticker/Tag
US $0.08-$0.30       Rfid Uhf H3 9662 9654 Chip Inlay/Label/Sticker Tag (Asset Warehousing Tracking)
US $0.06-$0.08       50*50Mm Uhf Bibliotheek Boek Documenten Rfid Tag Sticker Label
US $0.09-$0.15       Alien H3 9662, Alien H3 9654, Alien H4 UHF RFID Inlay/Sticker/Label
US $0.06-$0.15       Gratis Sample Lange Bereik H3 Passieve Uhf Herbruikbare Rfid Sticker Tag Label Voor Asset Tracking
US $0.06-$0.13       Chenxin Apparel Management Custom Afdrukken Uhf Rfid Tag Rfid Kledingstuk Wassen Zorg Etiketten Voor Kleding
US $0.23-$0.25       Rfid Uhf Electronic Label Washing Cloth Washing Label Heat Resistant Rfid Label Flexible Clothing
US $0.09-$0.12       Hot Selling Passief Printable Inlay Sticker Tag Uhf Rfid Label Voor Magazijn Retail
US $0.03-$0.09       Global UHF RFID Label U7 RFID Tag Voor Bril Frames
US $0.06-$0.50       Factory price UHF RFID label/tag adhesive
US $0.06             LX-C90G Rfid Voorruit Tag Passieve Long Range Uhf Rfid Sticker Label Voor Auto Tol Tracking Voertuig Registratie Of Parking

Upvotes: 1

Related Questions