CODEWITHSUNDEEP
python-3.xbeautifulsoup
yashul
yashul

Reputation: 101

How to extract information using BeautifulSoup from a particular site

My objective is to extract info from site https://shopopenings.com/merchant-search after entering pin code of the respective area and copy all info from there. Whether outlet is opened or closed. There has to be loop.

Upvotes: 1

Views: 100

Answers (2)

n1colas.m
n1colas.m

Reputation: 3989

This site has an underlying API that you can use to get JSON responses. To find the endpoints and what is expected as request and response you can use the Mozilla developer tools and Chrome devtools under network.

import json
import requests

SEARCH_ADDRESS = "California City, CA 93505"

urlEndpoint_AutoComplete = "https://shopopenings.com/api/autocomplete"
urlEndpoint_Search = "https://shopopenings.com/api/search"

search_Location = {"type":"address", "searchText":SEARCH_ADDRESS, "language":"us"}

locations = requests.post(urlEndpoint_AutoComplete, data=search_Location)

local = json.loads(locations.text)[0] # get first address
local["place_address"] = local.pop("name") # fix key name for next post request
local["place_id"] = local.pop("id") # fix key name for next post request

local["shopTypes"] = ["ACC", "ARA", "AFS", "AUT", "BTN", "BWL", "BKS", "AAC",
 "CEA", "CSV", "DPT", "DIS", "DSC", "DLS", "EQR", "AAF", "GHC", "GRO", "HBM",
 "HIC", "AAM", "AAX", "MER", "MOT", "BMV", "BNM", "OSC", "OPT", "EAP", "SHS",
  "GSF", "SGS", "TEV", "TOY", "TAT", "DVG", "WHC", "AAW"]

local["range"] = 304.8
local["language"] = "us"

results = requests.post(urlEndpoint_Search, data=local)
print(json.loads(results.text))

{'center': {'latitude': 35.125801, 'longitude': -117.9859038},
 'range': '304.8',
 'merchants': [{'mmh_id': '505518130',
   'latitude': 35.125801,
   'longitude': -117.9859,
   'shopName': 'Branham M Branham Mtr',
   'shopFullAddressString': 'California City, CA',
   'isOpen': False,
   'nfc': False,
   'shopType': 'AUT',
   'distance': 0.34636329,
   'country': 'USA'},
  {'mmh_id': '591581670',
   'latitude': 35.125442,
   'longitude': -117.986083,
   'shopName': 'One Stop Market',
   'shopFullAddressString': '7990 California City Blvd, California City, CA 93505-2518',
   'isOpen': True,
   'nfc': True,
   'shopType': 'AFS',
   'distance': 43.04766933,
   'country': 'USA'},
   ...
   ...

Upvotes: 1

theseeker
theseeker

Reputation: 1580

I think use selenium to control the Navigation and the entering of the Pin then use BeautifulSoup to work with the Page Source after you action. Here is the documentation it's easy enough to get you started.

Selenium -- https://selenium-python.readthedocs.io/

BeautifulSoup -- https://readthedocs.org/projects/beautiful-soup-4/

Enjoy!!

Upvotes: 0

Related Questions