CODEWITHSUNDEEP
pythonpython-3.xseleniumweb-scrapingpython-requests
Abhishek Rai
Abhishek Rai

Reputation: 2227

python Webscrape requests vs selenium

I am not really clear about the response I am getting here. It looks incomplete. I was wondering if this is the wrong approach and should switch to selenium. I am trying to get all the menu items along with the prices and the addons here. I was just checking the response. Any guidance would be immensely helpful. Thanks

import requests
from bs4 import BeautifulSoup


url = 'https://www.swiggy.com/dapi/menu/v4/full?lat=12.9715987&lng=77.5945627&menuId=41100'
page = requests.get(url).json()
for k, v in page.items():
    print(v)

Upvotes: 0

Views: 185

Answers (2)

Ahmet Yıldırım
Ahmet Yıldırım

Reputation: 56

Try this:

import requests

url = 'https://www.swiggy.com/dapi/menu/v4/full?lat=12.9715987&lng=77.5945627&menuId=41100'
page = requests.get(url).json()
menu_items = page["data"]["menu"]["items"]

for k, v in menu_items.items():
    print("Name: {}".format(v["name"]))
    print("Description: {}".format(v["description"]))
    print("Price: {}".format(v["price"]))
    if "addons" in v.keys():
        for i in v["addons"]:
            print("\t{}".format(i["group_name"]))
            for j in i["choices"]:
                print("\t\tName: {}".format(j["name"]))
                print("\t\tPrice: {}".format(j["price"]))
                print()
    print()

Upvotes: 1

HedgeHog
HedgeHog

Reputation: 25072

What is going on?

Think it is not a question of request vs selenium for web-scraping cause all that you are doing is requesting an api and getting a json response.

And your code works fine the way it is written, but what you expect to do is something other, so you have to deal with iterating dict in another way.

This will do the trick:

page['data']['menu']['items'].items()

Example

import requests

url = 'https://www.swiggy.com/dapi/menu/v4/full?lat=12.9715987&lng=77.5945627&menuId=41100'
page = requests.get(url).json()

for k, v in page['data']['menu']['items'].items():
    print(v)

Output

{'id': 8194821, 'name': 'Egg Omelette', 'category': 'Quick Bites', 'description': '', 'cloudinaryImageId': '', 'recommended': 0, 'inStock': 0, 'isVeg': 0, 'enabled': 1, 'displayOrder': 0, 'price': 8500, 'variants_new': {'exclude_list': [], 'variant_groups': []}, 'addons': [{'group_id': 21778839, 'group_name': 'Addons', 'choices': [{'id': 17471659, 'name': 'Watermelon Juice', 'price': 10000, 'inStock': 0, 'isVeg': 1, 'order': 1, 'default': 0}, {'id': 17471658, 'name': 'Coke (750 ml)', 'price': 6000, 'inStock': 1, 'isVeg': 1, 'order': 1, 'default': 0}], 'maxAddons': -1, 'maxFreeAddons': -1, 'minAddons': 0, 'order': 1}, {'group_id': 21778838, 'group_name': 'Beverage Addons', 'choices': [{'id': 17471658, 'name': 'Coke (750 ml)', 'price': 6000, 'inStock': 1, 'isVeg': 1, 'order': 1, 'default': 0}], 'maxAddons': -1, 'maxFreeAddons': -1, 'minAddons': 0, 'order': 1}], 'cropChoices': 2, 'itemScore': 0, 'itemDiscount': 0, 'isPopular': 0, 'restId': '41100', 'showMC': 0, 'ribbon': {'text': 'Must Try', 'textColor': '#ffffff', 'topBackgroundColor': '#d53d4c', 'bottomBackgroundColor': '#b02331'}, 'attributes': {'portionSize': '', 'vegClassifier': 'EGG', 'accompaniments': ''}, 'itemNudgeType': ''}
{'id': 8194822, 'name': 'Paneer Sholey', 'category': 'Starters', 'description': 'Fried cottage cheese cubes, tossed in signature spices & curry leaves garnish.', 'cloudinaryImageId': '', 'recommended': 0, 'inStock': 0, 'isVeg': 1, 'enabled': 1, 'displayOrder': 0, 'price': 22000, 'variants_new': {'exclude_list': [], 'variant_groups': []}, 'cropChoices': 2, 'itemScore': 0, 'itemDiscount': 0, 'isPopular': 0, 'restId': '41100', 'showMC': 0, 'attributes': {'portionSize': '', 'vegClassifier': 'VEG', 'accompaniments': ''}, 'itemNudgeType': ''}
...

Upvotes: 1

Related Questions