CODEWITHSUNDEEP
pythonpython-2.7web-scrapingbeautifulsoup
JasonC
JasonC

Reputation: 25

python web scraper - what have I done wrong?

I am a newbie and am building a web scraper that will grab (and eventually export to csv) all the UK McDonalds addresses, postcodes and phone numbers. I am using an aggregator instead of the McDonalds website.

https://www.localstore.co.uk/stores/75639/mcdonalds-restaurant/

I have borrowed and repurposed some code:

from bs4 import BeautifulSoup
from urllib2 import urlopen

BASE_URL = "https://www.localstore.co.uk/stores/75639/mcdonalds-restaurant/"

def get_category_links(section_url):
    html = urlopen(section_url).read()
    soup = BeautifulSoup(html, "lxml")
    boccat = soup.find("tr")
    category_links = [BASE_URL + tr.a["href"] for tr in boccat.findAll("h2")]
    return category_links

def get_restaurant_details(category_url):
    html = urlopen(category_url).read()
    soup = BeautifulSoup(html, "lxml")
    streetAddress = soup.find("span", "streetAddress").string
    addressLocality = [h2.string for h2 in soup.findAll("span", "addressLocality")]
    addressRegion = [h2.string for h2 in soup.findAll("span", "addressRegion")]
    postalCode = [h2.string for h2 in soup.findAll("span", "postalCode")]
    phoneNumber = [h2.string for h2 in soup.findAll("td", "b")]
    return {"streetAddress": streetAddress,
            "addressLocality": addressLocality,
            "postalCode": postalCode,
            "addressRegion": addressRegion,
            "phoneNumber": phoneNumber}

I don't think I have grabbed the data - as when I run the following line:

print(postalCode)

or 

print(addressLocality)

I get the following error 

NameError: name 'postalCode' is not defined

any idea with what i'm doing wrong?

Upvotes: 0

Views: 211

Answers (1)

Kristopher Ruzic
Kristopher Ruzic

Reputation: 31

As others have commented, you need to actually call your functions first off.

Do something like this

if __name__ == '__main__':
    res = "https://www.localstore.co.uk/store/329213/mcdonalds-restaurant/london/"
    print(get_restaurant_details(res)["postalCode"])

after your two functions. I just went on the site and got a URL that would work for your program, but I never actually tested it. The main problem you have right now is that you aren't actually doing anything. You need to call a function!

Upvotes: 3

Related Questions