CODEWITHSUNDEEP
pythonjsondictionaryevalabstract-syntax-tree
TJ Biddle
TJ Biddle

Reputation: 6484

Using ast.literal_eval on a nested dictionary

I'm using ast.literal_eval to change the data I receive from json.loads() into a Python dictionary; however if I should just be going about this an entirely different way - feel free to point that out as well.

# Authentication
buf = StringIO.StringIO()
c = pycurl.Curl()
c.setopt(c.URL, "https://kippt.com/api/account")
c.setopt(c.WRITEFUNCTION, buf.write)
c.setopt(c.HTTPHEADER, header)
c.perform()

result = buf.getvalue()
buf.close()

print result

# Printing Output
data_string = json.dumps(result)
jsonload = json.loads(data_string)
jsondict = ast.literal_eval(jsonload)

Currently it works properly with a one line JSON return, eg:

{"username": "my_username", "api_token": "my_api_token"}

and I can get the values properly with:

print jsondict['username']
print jsondict['api_token']

The part where I'm having an issue with is when the data is nested, such as:

{"meta": {"next": null, "total_count": 6, "previous": null, "limit": 20, "offset": 0}, "objects": [{"rss_url": "https://kippt.com/feed/username_here/stuff_here/cool-stuff", "updated": "1339003710", "title": "Cool Stuff", "created": "1339001514", "slug": "cool-stuff", "id": 54533, "resource_uri": "/api/lists/54533/"}, {"rss_url": "https://kippt.com/feed/username_here/stuff_here/programming", "updated": "1339003479", "title": "Programming", "created": "1339001487", "slug": "programming", "id": 54532, "resource_uri": "/api/lists/54532/"}, {"rss_url": "https://kippt.com/feed/username_here/stuff_here/android", "updated": "1339003520", "title": "Android", "created": "1339000936", "slug": "android", "id": 54530, "resource_uri": "/api/lists/54530/"}, {"rss_url": "https://kippt.com/feed/username_here/stuff_here/chrome", "updated": "1339000931", "title": "Chrome", "created": "1339000412", "slug": "chrome", "id": 54529, "resource_uri": "/api/lists/54529/"}, {"rss_url": "https://kippt.com/feed/username_here/stuff_here/inbox", "updated": "1338946730", "title": "Inbox", "created": "1338945940", "slug": "inbox", "id": 54432, "resource_uri": "/api/lists/54432/"}, {"rss_url": "https://kippt.com/feed/username_here/stuff_here/read-later", "updated": "1338945940", "title": "Read Later", "created": "1338945940", "slug": "read-later", "id": 54433, "resource_uri": "/api/lists/54433/"}]}

When I use the same code (Exchange URL for /api/lists) then I get the following error upon running the script:

Traceback (most recent call last): File "kippt.py", line 48, in jsondict = ast.literal_eval(jsonload) File "/usr/local/lib/python2.7/ast.py", line 80, in literal_eval return _convert(node_or_string) File "/usr/local/lib/python2.7/ast.py", line 63, in _convert in zip(node.keys, node.values)) File "/usr/local/lib/python2.7/ast.py", line 62, in return dict((_convert(k), _convert(v)) for k, v File "/usr/local/lib/python2.7/ast.py", line 63, in _convert in zip(node.keys, node.values)) File "/usr/local/lib/python2.7/ast.py", line 62, in return dict((_convert(k), _convert(v)) for k, v File "/usr/local/lib/python2.7/ast.py", line 79, in _convert raise ValueError('malformed string') ValueError: malformed string

Any help would be appreciated. Thanks!

Edit - Answer below:

Looks like my first input could have been interpretted as Python syntax, which is where my fault lay as I technically wasn't doing it the right way to begin with even with that.

I now just want to json.loads() my result from cURL rather than doing the screwy stuff I was doing before.

eg:

buf = StringIO.StringIO()
c = pycurl.Curl()
c.setopt(c.URL, "https://kippt.com/api/lists")
c.setopt(c.WRITEFUNCTION, buf.write)
c.setopt(c.HTTPHEADER, header)
c.perform()

result = buf.getvalue()
buf.close()

print result

# Printing Output
jsonload = json.loads(result)
print jsonload['meta']['total_count'] # Gets the total_count item in the meta object.

Upvotes: 4

Views: 11142

Answers (2)

Prakhar
Prakhar

Reputation: 1

I came up with one scenario where in I wanted to use json.normalize in pandas but values were of str type. using ast.literal_eval, i type casted

You can use ast.literal_eval like below - df[column_name] = df[column_name].apply(ast.literal_eval)

this will convert str to dict. for eg. df = [{'A': "{'value': '1'}", 'B': "{'value': '2'}"}]

after applying literal eval - df = [{'A': {'value': '1'}, 'B': {'value': '2'}}]

Upvotes: 0

David Wolever
David Wolever

Reputation: 154454

ast.literal_eval has no problem with nested dictionaries:

>>> ast.literal_eval("{'a': {'b':'c'}}")
{'a': {'b': 'c'}}

ast.literal_eval is breaking because the data are, in fact, JSON… And JSON is not valid Python. Specifically, null is not a valid Python literal.

Why not just use json.loads() to load the data?

Upvotes: 11

Related Questions