CODEWITHSUNDEEP
pythonweb-scrapingsessionpython-requestsshopee
Rafael Ribeiro
Rafael Ribeiro

Reputation: 47

Scraping products from shopee using python requests returns 90309999

Since 2022, I was able to scrap shopee listings using python requests with headers User-Agent and From only. However, since last week, I've been receiving error code number 90309999 from {"is_customized":false,"is_login":false,"action_type":2,"error":90309999,"tracking_id":"583f2b10-6b13-4402-8b3b-a6b8f0fead62"}

Also, I've checked this stackoverflow thread and added headers x-api-source and af-ac-enc-dat, but isn't working. Any suggestions?

from requests import Session

base_url = 'https://shopee.com.br/search?filters=9&keyword=funko&locations=Nacional&noCorrection=true&page=0&sortBy=relevancy'

headers = {
    'User-Agent': 'Googlebot',
    'From': '',
    'af-ac-enc-dat': '',
    'x-api-source': 'pc',
}

with Session() as s:
    s.get(base_url, headers=headers)

Don't want to use Selenium or Scrapy for this task, thanks!

EDIT: Also, I've insert null to af-ac-enc-dat but same results are shown.

Upvotes: 1

Views: 4079

Answers (2)

Shih Ying Jay
Shih Ying Jay

Reputation: 21

It is impossible to scrape shopee without using selenium now because shopee requires you to put headers at least with dynamic values of ( Af-Ac-Enc-Dat / Af-Ac-Enc-Sz-Token/ Cookie/ Referer(not sure) / X-Sap-Ri/X-Sap-Sec)

which you can only see while using selenium or open chrome browser. I paste the values manually to my request headers in python and it works to get the product detail successfully as json file.

However, the first problem is shopee can detect we using selenium most of time and redirect you to slider captcha page which you couldnt even slide at all manually.

even though we try to hide the selenium features. you can check if you hide selenium good enough with this website ([https://bot.sannysoft.com/][1]). Once you solve the selenium detect,the second problem is how to retrieve all of those dynamic values and parse them to our python as new headers while we go through every page with selenium. Once the second problem can be solved, we can scrape shopee with python request method again.

Otherwise, if only solve the first problem to avoid being detected, you can only use selenium to scrape shopee slowly.

BTW, I am using shopee taiwan,not sure if we are talking about the same shopee area.

Upvotes: 2

datawookie
datawookie

Reputation: 6564

Do you need to scrape using the Googlebot user agent? How about just using a Mozilla user agent?

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:122.0) Gecko/20100101 Firefox/122.0",
    "From": "",
    "af-ac-enc-dat": "",
    "x-api-source": "pc",
}

params = {
    "filters": "9",
    "keyword": "funko",
    "locations": "Nacional",
    "noCorrection": "true",
    "page": "0",
    "sortBy": "relevancy",
}

with requests.Session() as session:
    response = session.get("https://shopee.com.br/search", params=params, headers=headers)
    print(response)

Upvotes: 0

Related Questions