CODEWITHSUNDEEP
pythonjsonscrapy
Jeroen
Jeroen

Reputation: 460

Decoding HTML character entities in JSON

When I do this:

s = response.xpath('//meta[@id="_bootstrap-neighborhood_card"]').extract()

what I get back is:

<meta content='{"hosting":{"id":2256573,"offset_lat":39.04258923718809,"offset_lng":-95.69083697887662},"map_url":"https://maps.googleapis.com/maps/api/staticmap?markers=%2C&amp;size&amp;zoom=14","place_recommendations":[],"neighborhood_breadcrumb_details":[{"link_text":"Southwest Fillmore Street,","search_text":"Southwest Fillmore Street Topeka, KS","link":"&lt;span&gt;Southwest Fillmore Street,&lt;/span&gt;","link_route":"/s/Southwest-Fillmore-Street-Topeka--KS"},{"link_text":"Topeka,","search_text":"Topeka, KS","link":"&lt;span&gt;Topeka,&lt;/span&gt;","link_route":"/s/Topeka--KS"},{"link_text":"Kansas,","search_text":"Kansas, United States","link":"&lt;span&gt;Kansas,&lt;/span&gt;","link_route":"/s/Kansas--United-States"},{"link_text":"United States","search_text":"United States","link":"&lt;span&gt;United States&lt;/span&gt;","link_route":"/s/United-States"}],"neighborhood_basic_info":null,"neighborhood_localized_name":null,"user_info":{"user_image":"&lt;img alt=\"Elizabeth\" data-pin-nopin=\"true\" height=\"90\" src=\"https://a0.muscache.com/im/users/9199018/profile_pic/1380782460/original.jpg?aki_policy=profile_x_medium\" title=\"Elizabeth\" width=\"90\" /&gt;"}}' id="_bootstrap-neighborhood_card">

Which is clearly JSON but it's encoded (as you can see). I tried urllib.unquote but that throws an error. AttributeError: 'list' object has no attribute 'split'

I was hoping to not have to resort to using a regex to do the URL decoding. What can I do (besides using a regex) to make this valid JSON?

Upvotes: 0

Views: 2242

Answers (2)

mhawke
mhawke

Reputation: 87094

You can decode using json.loads(), however, you need to get at the JSON string contained in the content attribute of <meta> tag.

You can make multiple calls to xpath() to drill into the attributes of the selected tag:

meta = response.xpath('//meta[@id="_bootstrap-neighborhood_card"]')
content = meta.xpath('@content').extract_first()
data = json.loads(content)

Or you can do it in one go:

content = response.xpath('//meta[@id="_bootstrap-neighborhood_card"]').xpath('@content').extract_first()
data = json.loads(content)
from pprint import pprint
pprint(data)

Output

{u'hosting': {u'id': 2256573,
              u'offset_lat': 39.04258923718809,
              u'offset_lng': -95.69083697887662},
 u'map_url': u'https://maps.googleapis.com/maps/api/staticmap?markers=%2C&size&zoom=14',
 u'neighborhood_basic_info': None,
 u'neighborhood_breadcrumb_details': [{u'link': u'Southwest Fillmore Street,',
                                       u'link_route': u'/s/Southwest-Fillmore-Street-Topeka--KS',
                                       u'link_text': u'Southwest Fillmore Street,',
                                       u'search_text': u'Southwest Fillmore Street Topeka, KS'},
                                      {u'link': u'Topeka,',
                                       u'link_route': u'/s/Topeka--KS',
                                       u'link_text': u'Topeka,',
                                       u'search_text': u'Topeka, KS'},
                                      {u'link': u'Kansas,',
                                       u'link_route': u'/s/Kansas--United-States',
                                       u'link_text': u'Kansas,',
                                       u'search_text': u'Kansas, United States'},
                                      {u'link': u'United States',
                                       u'link_route': u'/s/United-States',
                                       u'link_text': u'United States',
                                       u'search_text': u'United States'}],
 u'neighborhood_localized_name': None,
 u'place_recommendations': [],
 u'user_info': {u'user_image': u''}}

Upvotes: 1

alecxe
alecxe

Reputation: 473933

Get the value of the content attribute and load it via json.loads():

>>> import json
>>> content = response.xpath('//meta[@id="_bootstrap-neighborhood_card"]/@content').extract_first()
>>> json.loads(content)

Note that you also need to use extract_first() instead of extract() to get a string value and not a list.

Upvotes: 2

Related Questions