CODEWITHSUNDEEP
python-3.xseleniumweb-scrapingbeautifulsouppython-requests
bsaoptima
bsaoptima

Reputation: 175

How to scrape dynamic content with Selenium in Python?

I'm trying to scrape the following page: https://www.heo.co.uk/uk/en/product/FRYU40156

I understand that this is a dynamic website so using the classic Request & Beautiful Soup combo might not work. I'm trying using Selenium and I do get more information by I can't seem to get the following: image of the bit I'm trying to get

I've tried to use By.TAG_NAME and 'h1' but it does not work.

Any ideas of how I can get the header ?

Thanks !

Upvotes: 1

Views: 67

Answers (1)

KunduK
KunduK

Reputation: 33384

If you go to network tab you will get the following api

enter image description here

https://www.heo.co.uk/api/article

Use the api to get the header value

import requests
payload={
  "articleNumber": "FRYU40156",
  "language": "en"
}
r=requests.post("https://www.heo.co.uk/api/article",data=payload).json()
print(r['article']['localization']['deName'])

output:

Jujutsu Kaisen 0: The Movie Hikkake PVC Statue Satoru Gojo 10 cm

Upvotes: 1

Related Questions