Convert Unicode to Dictionary

Question

Version: Python 2.7.10.

I have the following:

r = requests.post(url = API_ENDPOINT, headers = headers, data = data)
print(r.text)
print(type(r.text))
print(r.text[2])

The output

{"type":"quote","symbol":"SPY","bid":266.52,"bidsz":5,"bidexch":"P","biddate":"1513040398000","ask":266.55,"asksz":100,"askexch":"P","askdate":"1513040399000"}
{"type":"trade","symbol":"SPY","exch":"P","price":"266.31","size":"0","cvol":"83077533","date":"1513040400000","last":"266.31"}
{"type":"summary","symbol":"SPY","open":"265.58","high":"266.38","low":"265.4793","prevClose":"265.51","close":"266.31"}  
<type 'unicode'>
b

I would like to get the output "SPY".

I added the following:

new = simplejson.loads(r.text)
print(new)

Now, I get the following:

Traceback (most recent call last):
  File "example.py", line 63, in <module>
    new = simplejson.loads(r.text)
  File "/Library/Python/2.7/site-packages/simplejson/__init__.py", 
line 518, in loads
    return _default_decoder.decode(s)
  File "/Library/Python/2.7/site-packages/simplejson/decoder.py", 
line 373, in decode
    raise JSONDecodeError("Extra data", s, end, len(s))
simplejson.errors.JSONDecodeError: Extra data: line 1 column 160 - 
line 1 column 407 (char 159 - 406)

I changed:

r = requests.post(url = API_ENDPOINT, headers = headers, data = 
data).json()

Now, I get:

Traceback (most recent call last):
  File "example.py", line 51, in <module>
    r = requests.post(url = API_ENDPOINT, headers = headers, data = 
data).json()
  File "/Library/Python/2.7/site-packages/requests/models.py", line 
884, in json
    self.content.decode(encoding), **kwargs
  File "/Library/Python/2.7/site-packages/simplejson/__init__.py", 
line 518, in loads
    return _default_decoder.decode(s)
  File "/Library/Python/2.7/site-packages/simplejson/decoder.py", 
line 373, in decode
    raise JSONDecodeError("Extra data", s, end, len(s))
simplejson.errors.JSONDecodeError: Extra data: line 1 column 160 - 
line 1 column 407 (char 159 - 406)

I added:

new = json.dumps(r.text)
print(new)
print(type(new))

Now, output is:

"{\"type\":\"quote\",\"symbol\":\"SPY\",\"bid\":266.52,\"bidsz\":5,\"bidexch\":\"P\",\"biddate\":\"1513040398000\",\"ask\":266.55,\"asksz\":100,\"askexch\":\"P\",\"askdate\":\"1513040399000\"}
{\"type\":\"trade\",\"symbol\":\"SPY\",\"exch\":\"P\",\"price\":\"266.31\",\"size\":\"0\",\"cvol\":\"83077533\",\"date\":\"1513040400000\",\"last\":\"266.31\"}
{\"type\":\"summary\",\"symbol\":\"SPY\",\"open\":\"265.58\",\"high\":\"266.38\",\"low\":\"265.4793\",\"prevClose\":\"265.51\",\"close\":\"266.31\"}"

<type 'str'>

If I do:

for line in r.text.splitlines():
    d = json.loads(line)

I get:

Traceback (most recent call last):
File "example.py", line 54, in <module>
d = json.loads(line)
File "/System/Library/Frameworks/Python.
framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in 
loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/
Versions/2.7/lib/python2.7/json/decoder.py", line 369, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 158 - line 1 column 404 
(char 157 - 403)

So I tried:

for line in r.text.splitlines():
    print(line)
    print("\n\n")

And I got:

{"type":"quote","symbol":"SPY","bid":267.18,"bidsz":1,"bidexch":"P","biddate":"1513213200000","ask":267.22,"asksz":3,"askexch":"P","askdate":"1513213200000"}
{"type":"quote","symbol":"SPY","bid":267.18,"bidsz":1,"bidexch":"P","biddate":"1513213200000","ask":267.22,"asksz":3,"askexch":"P","askdate":"1513213200000"} 

So even though there should be two lines, it interprets everything as a single line.

How can I convert r.text to a dictionary?

Answer 1

I believe what you are receiving is JSON Lines format. Each line of the response is a single json string. I simulated the response:

import json

class r: pass
r.text = u'''\
{"type":"quote","symbol":"SPY","bid":266.52,"bidsz":5,"bidexch":"P","biddate":"1513040398000","ask":266.55,"asksz":100,"askexch":"P","askdate":"1513040399000"}
{"type":"trade","symbol":"SPY","exch":"P","price":"266.31","size":"0","cvol":"83077533","date":"1513040400000","last":"266.31"}
{"type":"summary","symbol":"SPY","open":"265.58","high":"266.38","low":"265.4793","prevClose":"265.51","close":"266.31"}
'''    

print(r.text)
print(type(r.text))
print(r.text[2])

# Parse JSON a line at a time:
for line in r.text.splitlines():
    d = json.loads(line)
    print d['symbol']

Output:

{"type":"quote","symbol":"SPY","bid":266.52,"bidsz":5,"bidexch":"P","biddate":"1513040398000","ask":266.55,"asksz":100,"askexch":"P","askdate":"1513040399000"}
{"type":"trade","symbol":"SPY","exch":"P","price":"266.31","size":"0","cvol":"83077533","date":"1513040400000","last":"266.31"}
{"type":"summary","symbol":"SPY","open":"265.58","high":"266.38","low":"265.4793","prevClose":"265.51","close":"266.31"}

<type 'unicode'>
t
SPY
SPY
SPY

Answer 2

You should be using this .json():

r.json()