CODEWITHSUNDEEP
pythonpython-3.xpython-requests
curious_nustian
curious_nustian

Reputation: 616

Unable to fetch JSON response

I am sending a GET request to this site https://www.everything5pounds.com/en/Womens/c/womens#/?q=&sort=newArrivals and response I get is page source (same as that browser renders)

But when I use networks tab in chrome I see the response to URL as JSON. It is strange that I am not able to get JSON response despite using "accept":"application/json".

Following is code I am using.

import requests
from bs4 import BeautifulSoup

headers = requests.utils.default_headers()
headers.update({
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
    'accept':'application/json'
})
url = 'https://www.everything5pounds.com/en/Womens/c/womens#/?q=&sort=newArrivals'
response = requests.get(url)
content = BeautifulSoup(response.content,'lxml')
print(content)

Kindly correct me if I am doing something wrong, or otherwise explain the reason behind this.

Upvotes: 0

Views: 72

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195408

You don't have your URL correct:

import json
import requests
from pprint import pprint

url = 'https://www.everything5pounds.com/en/Womens/c/womens/results/?q=&sort=newArrivals'

data = json.loads(requests.get(url).text)
# You can get json also directly, no need to import json library:
# data = requests.get(url).json()


pprint(data)

Prints:

{'currentQuery': ':newArrivals',
 'pagination': {'currentPage': 0,
                'numberOfPages': 458,
                'pageSize': 24,
                'sort': 'newArrivals',
                'totalNumberOfResults': 10973},
 'results': [{'availableForPickup': None,
              'availableInCurrentStore': None,
              'averageRating': 5.0,
              'badgeCode': None,
              'badgeUrl': None,
              'baseOptions': None,
              'baseProduct': None,
              'baseProductUrl': None,
              'categories': None,
              'categoryUrl': None,
              'classifications': None,
              'cleanUrl': '/Tie-Up-Cold-Shoulder-Dip-Hem-Dress/p/659773',
              'code': '659773',

...and so on.

Upvotes: 1

Related Questions