CODEWITHSUNDEEP
web-scrapingbeautifulsouplocation
John_Muir
John_Muir

Reputation: 119

Scraping location using beautiful soup

I am super new to beautifulsoup, I have done tons of online videos and now I am adventuring to my first project.

Anyway, my goal is to scrape the location of https://www.mastermindtoys.com/apps/store/find-a-store. All the locations are under one class "clearfix large-container". I am wondering how do I pull out the information of the address from all "address-sec". "address-sec" being the class that is under "clearfix large-container". If anyone has a video that goes inline or at advice that would be great!

Upvotes: 1

Views: 628

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195553

The data you see is loaded from external source, so beautifulsoup doesn't see it. But you can simulate Ajax requests with requests module to load it:

import json
import requests
from bs4 import BeautifulSoup


url = "https://bopis.mastermindtoys.com/ajax/getstorelocation.php"

headers = {
    "Referer": "https://www.mastermindtoys.com/",
}

payload = {
    "doAction": "getNearStoreData",
    "latitude": "",
    "longitude": "",
    "flagDisplayFirstRecord": "Y",
    "shopifyCustomerId": "",
    "selectedStoreInCookies": "",
}

data = requests.post(url, headers=headers, data=payload).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for i, d in enumerate(data["details"], 1):
    print(
        "{:<3} {:<25} {:<30} {:<30}".format(
            i, d["name"], d["address1"], d["address2"]
        )
    )

Prints:

1   Abbotsford                32470 South Fraser Way         Clearbrook Town Square        
2   Langley                   20085 Langley Bypass           Langley Centre                
3   Coquitlam                 3000 Lougheed Hwy              Westwood Mall                 
4   Tsawwassen                5000 Canoe Pass Way            Tsawwassen Mills              
5   Langford                  2945 Jacklin Road              Westshore Town Centre         
6   Prince George             3034 Recplace Drive            River Point Landing           
7   Westhills                 600 Stewart Green SW           Westhills Town Centre         
8   Shawnessy                 16061 MacLeod Trail SE         Boulevard Shopping Centre     
9   McKenzie Towne            4307 130 Ave SE                South Trail Crossing          
10  Country Hills             450 Country Hills Blvd NE      Country Hills Town Centre     
11  Lethbridge                1250 2A Avenue North                                         
12  Red Deer                  5250 22nd Street                                             
13  Grande Prairie            11521 Westgate Drive           Westgate Centre               
14  Windermere                5236 Windermere Boulevard NW   Currents of Windermere        
15  Terra Losa                9752 170 Street                Terra Losa Centre             
16  South Edmonton            2004 99th Street NW                                          
17  St. Albert                398 St. Albert Trail           Mission Hill Plaza            
18  Sherwood Park             222 Baseline Road              Sherwood Park Baseline Village Plaza
19  Medicine Hat              1840 Strachan Road SE                                        
20  Saskatoon South           3011 Clarence Avenue South     Unit 170                      
21  Preston Crossing          1718 Preston Avenue North                                    
22  Regina East               2820 East Quance Street                                      
23  Kenaston                  1655 Kenaston Boulevard                                      
24  Kildonan                  1561 Regent Avenue West        Kildonan Mall                 
25  Sudbury                   1332 Kingsway Boulevard                                      
26  Windsor                   3175 Howard Avenue             Roundhouse Centre             
27  Owen Sound                1969 16th Street East                                        
28  London North              685 Fanshawe Park Road West                                  
29  London South              3120 Wonderland Road South     Wonderland Plaza              
30  Waterloo                  405 King Street North                                        
31  Barrie                    488 Bayfield Street            North Supercentre             
32  Cambridge                 612 Hespeler Road                                            
33  Guelph                    370 Stone Road West            Stone Square Centre           
34  Milton                    1079 Maple Avenue                                            
35  Newmarket                 18267 Yonge Street                                           
36  Aurora                    650 Wellington Street East     Aurora Gateway Centre         
37  Ancaster                  821 Golf Links Road            Meadowlands Entertainment Centre
38  Vaughan                   16 Famous Avenue               Colossus Centre               
39  Hurontario                26 Eglinton Avenue West                                      
40  Burlington                1035 Brant Street                                            
41  Oakville                  1011 Upper Middle Road East    Upper Oakville Shopping Centre
42  Richmond Hill             9350 Yonge Street                                            
43  Mississauga South         970 Southdown Road             Clarkson Crossing Centre      
44  Etobicoke                 4242 Dundas Street West        Kingsway Mills Shopping Centre
45  Hamilton                  2180 Rymal Road East                                         
46  Bayview Village           2901 Bayview Avenue            Bayview Village Shopping Centre
47  Yonge                     3350 Yonge Street                                            
48  Markham                   8555 McCowan Rd                                              
49  Mount Pleasant            639 Mount Pleasant Road                                      
50  Rosedale                  1133 Yonge Street                                            
51  The Beach                 2134 Queen Street East                                       
52  Pickering                 1355 Kingston Road             Pickering Town Centre         
53  Ajax                      170 Kingston Road East                                       
54  North Oshawa              991 Taunton Road East                                        
55  St. Catharines            210 Glendale Avenue                                          
56  Peterborough              995 Lansdowne Street West                                    
57  Kanata                    145 Roland Michener Drive      Kanata Centrum Plaza          
58  Kingston                  2511 Princess Street                                         
59  Barrhaven                 71 Marketplace Avenue          Chapman Mills Marketplace     
60  St. Laurent               1091 St Laurent Boulevard                                    
61  Orléans                   1315 Place d'Orleans Drive     Place d'Orléans               
62  Fredericton               38 Trinity Avenue                                            
63  Saint John                70 Consumers Drive             Westmorland Shopping Centre   
64  Moncton                   84 Wyse Street                 Mapleton Centre               
65  Halifax                   201 Chain Lake Drive                                         
66  Dartmouth                 29 Countryview Drive           Dartmouth Crossing            
67  Sydney                    20 Sydney Port Access Road                                   
68  St. John's North          50 White Rose Drive

Upvotes: 2

Related Questions