CODEWITHSUNDEEP
pythonpython-3.xseleniumweb-scrapingpython-requests
Stephan Yazvinski
Stephan Yazvinski

Reputation: 578

Scrape target search results with python

I'm am trying to scrape search results on Target.

For example, let's go to the domain "https://www.target.com/s?searchTerm=lego+duplo"

And try to extract the product names, prices, and product ids.

I've tried selenium and I get asked to verify my identity. I've tried requests and I get the forbidden page. I've tried other libraries and I'm running out of ideas.

Ideally, there should be some javascript that loads the prices with JSON but I cant seem to find it. Any advice?

Upvotes: 1

Views: 615

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195438

The data is loaded from the external URL in Json format. You can use next example how to simulate the Ajax request:

import json
import requests

url = "https://www.target.com/s?searchTerm=lego+duplo"
headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"
}

api_url = "https://redsky.target.com/redsky_aggregations/v1/web/plp_search_v1"

params = {
    "key": "ff457966e64d5e877fdbad070f276d18ecec4a01",
    "channel": "WEB",
    "count": "24",
    "default_purchasability_filter": "false",
    "include_sponsored": "true",
    "keyword": "lego duplo",
    "offset": "0",
    "page": "/s/lego duplo",
    "platform": "desktop",
    "pricing_store_id": "3991",
    "useragent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",
    "visitor_id": "AAA",
}

data = requests.get(api_url, params=params).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for p in data["data"]["search"]["products"]:
    print(
        "{:<10} {}".format(
            p["price"]["current_retail"],
            p["item"]["product_description"]["title"],
        )
    )

Prints:

29.99      LEGO DUPLO Classic Brick Box First LEGO Set with Storage Box 10913
23.99      LEGO DUPLO Super Heroes Lab Marvel Avengers Toy 10921
32.99      LEGO DUPLO Creative Fun 10887
16.39      LEGO DUPLO My First Number Train 10847
14.99      LEGO DUPLO Large Green Building Plate 2304
16.39      LEGO DUPLO Disney Minnie Mouse's Birthday Party 10873
24.49      LEGO DUPLO Disney Ariel&#39;s Undersea Castle Building Toy; Princess Castle Under the Sea 10922
16.39      LEGO DUPLO Construction Truck &#38; Tracked Excavator Digger and Tipper Building Site Toy 10931
16.39      LEGO DUPLO Disney Frozen Toy Featuring Elsa and Olaf&#39;s Tea Party 10920
9.99       LEGO DUPLO My First Fire Helicopter and Police Car 10957
49.99      LEGO DUPLO Fire Station 10903
16.39      LEGO DUPLO Submarine Adventure Bath Toy Building Set for Toddlers with Toy Submarine 10910
4.99       LEGO DUPLO My First Space Rocket 30332 Building Kit
15.99      LEGO City Construction Bulldozer Building Set 60252
29.99      LEGO DUPLO Disney Mickey &#38; Minnie Birthday Train Kids&#39; Birthday Number Train Playset 10941
41.99      LEGO DUPLO Princess Frozen Ice Castle Toy Castle Building Set with Frozen Characters 10899
19.99      LEGO DUPLO My First Animal Train Pull-Along 10955
19.99      LEGO DUPLO Fire Truck 10901
9.89       LEGO Classic Bricks and Ideas 11001
29.99      LEGO DUPLO Jurassic World T. rex and Triceratops Dinosaur Breakout 10939
20.99      LEGO DUPLO Town Airport 10871
23.99      LEGO Classic Medium Creative Brick Box Building Toys for Creative Play, Kids Creative Kit 10696
9.99       LEGO DUPLO Police Bike 10900
19.99      LEGO DUPLO Town Farm Tractor &#38; Animal Care Building Toy 10950

Upvotes: 2

Related Questions