CODEWITHSUNDEEP
pythonregexweb-scrapingbeautifulsouplatitude-longitude
backpackerice
backpackerice

Reputation: 87

How to use beautifulsoup to scrape the Latitude/Longitude data from html page

I'm trying to scrape the latitude & longitude number from this website:

http://www.healthgrades.com/provider-search-directory/search?q=Dentistry&prof.type=provider&search.type=&method=&loc=New+York+City%2C+NY+&pt=40.71455%2C-74.007118&isNeighborhood=&locType=%7Cstate%7Ccity&locIsSolrCity=false

For each provider, if you look at the element, it looks like 

div class="listing" data-lat="40.66862" data-lng="-73.98574" data-listing="22"

How can I get the latitude and Longitude number here using beautifulsoup?

I tried to use regex in my script,

Below is my script - 

Geo = soup.find("div", class_="providerSearchResults")
print Geo.findAll("div", data-lat_= re.compile('[0-9.]'))

But I get this error message: "SyntaxError: keyword can't be an expression"

Besides, for each provider, the "div" part always changes it can be:

div class="listing" data-lat="40.66862" data-lng="-73.98574" data-listing="22"

or

div class="listingfirst" data-lat="40.66862" data-lng="-73.98574" data-listing="22"

or even

div class="listing enhancedlisting" data-lat="40.66862" data-lng="-73.98574" data-listing="22"

Upvotes: 2

Views: 1409

Answers (1)

Joe Young
Joe Young

Reputation: 5885

First a few requirements:

pip install requests
pip install BeautifulSoup
pip install lxml

latlongbs4.py:

import requests
from bs4 import BeautifulSoup

r = requests.get('http://www.healthgrades.com/provider-search-directory/search?q=Dentistry&prof.type=provider&search.type=&method=&loc=New+York+City%2C+NY+&pt=40.71455%2C-74.007118&isNeighborhood=&locType=%7Cstate%7Ccity&locIsSolrCity=false')
soup = BeautifulSoup(r.text, 'lxml')
latlonglist = soup.find_all(attrs={"data-lat": True, "data-lng": True})
for latlong in latlonglist:
    print latlong['data-lat'], latlong['data-lng']

EDIT: Removed class from the attrs dictionary.

Output:

(latlongbs4)macbook:latlongbs4 joeyoung$ python latlongbs4.py
40.71851 -74.00984
40.77536 -73.97707
40.71961 -74.00347
40.71395 -74.008
40.711614 -74.015901
40.724576 -74.001771
40.7175 -74.00087
40.71961 -74.00347
40.71766 -73.99293
40.71961 -74.00347
40.71848 -73.99648
40.709917 -74.009884
40.71553 -74.00977
40.71702 -73.996
40.71254 -73.99994
40.70869 -74.01164
40.70994 -74.00764
40.707325 -74.003982
40.7184 -74.00098
40.71373 -74.00812
40.710474 -74.009844
40.7175 -74.00087
40.727582 -73.894632
40.763469 -73.963106
40.724853 -73.841097

A few notes:

I used the attrs keyword with a dictionary because:

Some attributes, like the data-* attributes in HTML 5, have names that can’t be used as the names of keyword arguments:

You can use these attributes in searches by putting them into a dictionary and passing the dictionary into find_all() as the attrs argument:

Source: http://www.crummy.com/software/BeautifulSoup/bs4/doc/#the-keyword-arguments

Upvotes: 2

Related Questions