CODEWITHSUNDEEP
pythonbeautifulsoupweb-crawler
user12952651
user12952651

Reputation:

Got an empty list when scraping shopee.co.id using BeautifulSoup

I was trying to scrape names and sales of the top products on https://shopee.co.id/top_products using the BeautifulSoup package. But whatever I try, it always returns an empty list of products. Could you explain what happened under the hood? Here's my code:

import requests
from bs4 import BeautifulSoup as bs

headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',
    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'
}

shopee_url = 'https://shopee.co.id/top_products'
response = requests.get(shopee_url, headers=headers)
soup = bs(response.text, "html.parser")

products = soup.find_all("._3S8sOC _2QfAXF")
print(type(products))
print(products)

Thanks for reading this far. I have read some similar answers, but they don't seem to apply here.

Upvotes: 1

Views: 1184

Answers (1)

UWTD TV
UWTD TV

Reputation: 910

If you are willing to use selenium you can make the request with selenium:

from selenium import webdriver
from bs4 import BeautifulSoup as bs

headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',
    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'
}
driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe')
shopee_url = 'https://shopee.co.id/top_products'
driver.get(shopee_url)
driver.implicitly_wait(15)
response = driver.page_source
driver.close()
soup = bs(response, "html.parser")

products = soup.select(".shopee-item-card__lower-padding")

#print(products)

for product in products:
    name = product.select_one('.shopee-item-card__text-name span')
    print(name.get_text(strip=True))
    price = product.select_one('.shopee-item-card__current-price')
    print(price.text)

Note you need Firefox, geckodriver, and selenium installed.

Output:

Kuota Telkomsel Cek Dadakan
Rp300
Aigo 1GB 5Hari
Rp7.375
Paket Data Tri Mix Small 5GB
Rp4.500
Paket Data Tri Mix Small 2.75GB
Rp9.500
*PROMO MURAH* SURPRISE DEAL PAKET DATA INTERNET KUOTA TELKOMSEL SIMPATI AS LOOP OMG 20GB 30GB 50GB
Rp1.000
Kuota Telkomsel Cek Dadakan
Rp500
Paket Data Telkomsel 12GB
Rp98.000
Paket Data Tri AON Unlimited + 6GB
Rp58.000
KUOTA INTERNET PAKET DATA INTERNET TELKOMSEL KUOTA DATA FLASH Kartu Simpati As Loop (KUOTA MURAH)
Rp1.000
[MURAH] PAKET DATA TELKOMSEL ∣ KUOTA INTERNET ∣ PAKET DATA ∣ SIMPATI  ∣ AS  ∣ LOOOP
Rp1.000
Paket Data Tri AddOn 12GB
Rp48.000
Paket Data  Indosat Freedom U 7GB+20GB Apps / 30 Hari
Rp66.500
PAKET DATA TELKOMSEL ISI ULANG SIMPATI 14GB/27GB/37GB - AS 6,5GB/10GB/17GB/32GB - LOOP 3GB/8GB/15GB
Rp38.000
Paket Data Indosat Freedom Internet 4GB / 30 Hari
Rp24.000
Paket Data Indosat Freedom U 1GB+4.5GB Apps / 30 Hari
Rp24.000
KUOTA TELKOMSEL paket kuota data internet telkomsel termurah 52gb injek injeck kuota  promo
Rp1.000
Kuota Telkomsel Simpati 52GB 50GB 24jam Langsung Masuk
Rp1.000
Paket Data Telkomsel 50GB
Rp195.000
Paket Data Indosat Freedom Internet 10GB /30 Hari
Rp48.000
KUOTA TELKOMSEL paket data internet telkomsel kuota internet TELKOMSEL  termurah simpati as loop
Rp1.000
Paket Data Telkomsel Data 25.000
Rp23.500
VOCHER VOUCHER KUOTA AXIS AIGO 1GB 5HARI NASIONAL
Rp7.390
PAKET DATA KUOTA INDOSAT [DISKON 15%] TERMURAH
Rp1.000
VOUCHER Axis 1gb 5 hari 24 Jam KUOTA 1 GB
Rp7.500
Paket Data Indosat Freedom Combo 14GB, 30 Hari
Rp48.500
[PROMO] KUOTA BESAR PAKET OMG TELKOMSEL MURAH
Rp1.000
Paket Data Indosat Freedom Combo 8GB/ 30 Hari
Rp33.500
TELKOMSEL PEMBAYARAN RESELLER/ AGEN atau Penambahan Selisih Harga Produk ThalhahCell
Rp1.000
Aigo 3GB 15Hari
Rp18.500
Paket Data Indosat Freedom U 3GB+15GB Apps / 30 Hari
Rp49.900

Upvotes: 1

Related Questions