CODEWITHSUNDEEP
pythondictionarybeautifulsoupscreen-scraping
Amar
Amar

Reputation: 37

Using beautiful soup to extract from a dictionary

I am trying to extract some product and brand data from a website using beautiful soup and python.

I've managed to extract the html, below is an example:

 </a></div></div><div class="Product" data-sku="120348"><a class="Product-link thumb "
 data-dl-tracked='{"event":"productImpression","ecommerce":{"currencyCode":"GBP","impressions":
[{"name":"Kiehls Powerful Strength Bundle","price":"74.40","brand":"Kiehls","list":"Product Listing -
","position":-1,"id":"120348"}]}}' data-feelunique-datalayer-push='{"click": 
{"event":"productClick","ecommerce":{"currencyCode":"GBP","click":{"actionField":
{"list":"Product Listing -"},"products":[{"name":"Kiehls Powerful Strength 
Bundle","id":"120348","price":"74.40","brand":"Kiehls","position":-1}]}}}

and I'm using the below to get a list of dictionaries:

for i in soup.find_all("a" , {"class": "Product-link thumb "}): product.append(i.get("data-dl-tracked"))

but I'd like to be abe to get name and brand out of the dictionary. Any ideas?

Here's a link to the website incase it helps: https://www.feelunique.com/skin

Upvotes: 2

Views: 1896

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195438

You can use json module to parse the data stored in the HTML attribute. For example:

import json
import requests
from bs4 import BeautifulSoup


url = 'https://www.feelunique.com/skin'

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0',
    'Accept-Language': 'en-US,en;q=0.5'
}

soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')

for a in soup.select('a[data-feelunique-datalayer-push]'):
    data = json.loads(a['data-feelunique-datalayer-push'])
    # uncomment this to see all data:
    # print(json.dumps(data, indent=4))
    product = data['click']['ecommerce']['click']['products'][0]
    print('{:<30} {}'.format(product['brand'], product['name']))

Prints:

Powered by Feelunique          Megan Rose Lane Self-Care Kit
Wishful                        Wishful Honey Balm Moisturiser 55g
Kiehls                         Kiehls Ultra Facial Cream 50ml
Kiehls                         Kiehls Ultra Facial Cream 125ml
Charlotte Tilbury              Charlotte Tilbury Charlottes Magic Cream Moisturiser 50ml
CeraVe                         CeraVe Foaming Facial Cleanser 473ml
Wishful                        Wishful Yo Glow Enzyme Scrub 100ml
Kiehls                         Kiehls Powerful Strength Bundle
Kiehls                         Kiehls Midnight Recovery Concentrate Facial Oil 30ml
Estée Lauder                   Estée Lauder Advanced Night Repair Synchronized Multi-Recovery Complex 30ml
Kiehls                         Kiehls Rare Earth Deep Pore Cleansing Mask 125ml
Charlotte Tilbury              Charlotte Tilbury Morning Magic Skin Kit
Embryolisse                    Embryolisse Lait-Crème Concentré Moisturiser 75ml
Kiehls                         Kiehls Ultra Light Daily UV Defense SPF50 60ml
Kiehls                         Kiehls Creamy Eye Treatment with Avocado 28ml
Clinique                       Clinique Moisture Surge™ 72-Hour Auto-Replenishing Hydrator 50ml
The Inkey List                 The INKEY List Tranexamic Acid Night Treatment 30ml
Kiehls                         Kiehls Super Multi-Corrective Cream SPF 30 50ml
Kiehls                         Kiehls Creamy Eye Treatment with Avocado 14ml
Dermalogica                    Dermalogica Daily Microfoliant 75g
Clinique                       Clinique Take The Day Off Cleansing Balm 125ml
Estée Lauder                   Estée Lauder Advanced Night Repair Eye Supercharged Complex 15ml
Kiehls                         Kiehls Powerful-Strength Line-Reducing Concentrate 50ml
Grown Alchemist                Grown Alchemist Age-Repair Moisturiser Phyto-Peptide & White Tea Extract 60ml
Charlotte Tilbury              Charlotte Tilbury Charlottes Magic Lip Oil Crystal Elixir 8ml
The Inkey List                 The INKEY List Retinol Eye Cream 15ml
Lumene                         Lumene Nordic-C Glow Boost Essence Serum 30ml
Kiehls                         Kiehls Calendula Deep Cleansing Foaming Face Wash 230ml
Kiehls                         Kiehls Midnight Recovery Concentrate 50ml
Perricone MD                   Perricone MD Vitamin C Ester Citrus Brightening Cleanser 59ml
Charlotte Tilbury              Charlotte Tilbury Charlotte’s Cleanse, Hydrate & Glow Mini Facial Kit
Saturday Skin                  Saturday Skin No Bad Days Set
First Aid Beauty               First Aid Beauty Ultra Repair Hydrating Serum 50ml
Emma Hardie                    Emma Hardie Moringa Cleansing Balm 200g - 10th Anniversary Edition
DHC                            DHC Deep Cleansing Oil 200ml
Dermalogica                    Dermalogica Special Cleansing Gel 500ml
COSRX                          COSRX Acne Pimple Master Patch 24 Patches
Kiehls                         Kiehls Ultra Facial Cream SPF30 125ml
Charlotte Tilbury              Charlotte Tilbury Charlottes Magic Serum Crystal Elixir 30ml
Kiehls                         Kiehls Ultra Facial Favourites Gift Set

Upvotes: 4

Related Questions