CODEWITHSUNDEEP
pythonweb-scraping
Melissa A
Melissa A

Reputation: 39

crawling a webpage: empty list always return

I have some trouble with crowling this webpage, https://www.woolworths.com.au/shop/browse/drinks/cordials-juices-iced-teas/iced-teas, i tried many selectors, h1,title, price, and have always the same return : an empty list, can someone help me to extract products names? it's for a test, thank you very much

scrapy shell "https://www.woolworths.com.au/shop/browse/drinks/cordials-juices-iced-teas/iced-teas"

response.css(".tileList-title").extract() #for the title
response.css(".price-dollars::text").extract() # for the prices 
response.css(".shelfProductTile-descriptionLink") #for the products na:e

Upvotes: 0

Views: 262

Answers (1)

LuckyZakary
LuckyZakary

Reputation: 1191

The website has a private api that you can use to get the product data. I put part of the json response into a pandas dataframe. The api has a lot more info on the products that you can extract if you want.

Code

import json, requests, pandas as pd

data = json.loads(
    r'{"categoryId":"1_9573995","pageNumber":1,"pageSize":24,"sortType":"TraderRelevance","url":"/shop/browse/drinks/cordials-juices-iced-teas/iced-teas","location":"/shop/browse/drinks/cordials-juices-iced-teas/iced-teas","formatObject":"{\"name\":\"Iced Teas\"}","isSpecial":false,"isBundle":false,"isMobile":false,"filters":null}')
url = 'https://www.woolworths.com.au/apis/ui/browse/category'
r = requests.post(url, data=data)
api = json.loads(r.text)

to_df = {'Names': [], 'Prices': [], 'Cup Strings': []}
for item in api['Bundles']:
    for product_stuff in item['Products']:
        to_df['Names'].append(product_stuff['Name'])
        to_df['Prices'].append(product_stuff['Price'])
        to_df['Cup Strings'].append(product_stuff['CupString'])

df = pd.DataFrame(data=to_df)
print(df)

Output

                                        Names  Prices Cup Strings
0                 Lipton Lipton Ice Tea Peach    10.0  $3.57 / 1L
1                  Arizona Ice Tea With Peach     2.0  $4.00 / 1L
2                 Arizona Ice Tea Pomegranate     2.0  $4.00 / 1L
3        Arizona Green Tea With Honey Ice Tea     2.0  $4.00 / 1L
4            Lipton Ice Green Tea Light Lemon     3.9  $2.60 / 1L
5                        Lipton Ice Tea Lemon     3.9  $2.60 / 1L
6                        Lipton Ice Tea Lemon     1.9  $3.80 / 1L
7                  Lipton Ice Tea Light Peach     3.9  $2.60 / 1L
8                  Lipton Light Peach Ice Tea     1.9  $3.80 / 1L
9                    Lipton Ice Tea Raspberry     3.9  $2.60 / 1L
10                   Lipton Ice Tea Raspberry     1.9  $3.80 / 1L
11                       Lipton Ice Tea Peach     3.9  $2.60 / 1L
12                       Lipton Peach Ice Tea     1.9  $3.80 / 1L
13                       Lipton Green Ice Tea     1.9  $3.80 / 1L
14                    Arizona Iced Lemon Tea      2.0  $4.00 / 1L
15                       Lipton Ice Tea Peach     2.7  $7.71 / 1L
16              Lipton Ice Green Tea Original     3.9  $2.60 / 1L
17                       Lipton Ice Tea Mango     3.9  $2.60 / 1L
18  Lipton Ice Tea Matcha Ginger & Lemongrass     3.9  $2.60 / 1L
19                         Fuze Ice Tea Mango     2.1  $1.68 / 1L
20                         Fuze Ice Tea Peach     2.1  $1.68 / 1L

Upvotes: 1

Related Questions