Python JSON Serialize Dictionary<String,Object>

Question

I have a Dictionary with Key String (2019-10-28 13:21) and Value of Object (DataPoint)

import requests
import json
import time

symbol = "AAPL"

intraday_url = "https://www.alphavantage.co/query?function=TIME_SERIES_INTRADAY&symbol="+symbol+"&interval=1min&outputsize=full&apikey="+api_key
sma_url = "https://www.alphavantage.co/query?function=SMA&symbol="+symbol+"&interval=1min&time_period=180&series_type=open&apikey="+api_key
ema_url = "https://www.alphavantage.co/query?function=EMA&symbol="+symbol+"&interval=1min&time_period=15&series_type=open&apikey="+api_key
vwap_url = "https://www.alphavantage.co/query?function=VWAP&symbol="+symbol+"&interval=1min&apikey="+api_key
macd_url = "https://www.alphavantage.co/query?function=MACD&symbol="+symbol+"&interval=1min&series_type=open&apikey="+api_key
rsi_url = "https://www.alphavantage.co/query?function=RSI&symbol="+symbol+"&interval=1min&time_period=100&series_type=open&apikey="+api_key
adx_url = "https://www.alphavantage.co/query?function=ADX&symbol="+symbol+"&interval=1min&time_period=100&apikey="+api_key

class DataPoint:
    def __init__(self, time):
        # 2019-10-31 15:49:00 (original)
        # 2019-10-31 15:49 (formatted)
        formatted_time = time[0:len(time)-3]
        self.time = formatted_time
        self.open = None
        self.high = None
        self.low = None
        self.close = None
        self.volume = None
        self.sma = None
        self.ema = None
        self.vwap = None
        self.macd = None
        self.rsi = None
        self.adx = None

    def addIntraday(self,open,high,low,close,volume):
        self.open = open
        self.high = high
        self.low = low
        self.close = close
        self.volume = volume

    def addTechnical(self,technical,value):
        if technical == "SMA":
            self.sma = value
        elif technical == "EMA":
            self.ema = value
        elif technical == "VWAP":
            self.vwap = value
        elif technical == "MACD":
            self.macd = value
        elif technical == "RSI":
            self.rsi = value
        elif technical == "ADX":
            self.adx = value

def getIntraday(dictionary):
    url = intraday_url
    response = requests.get(url)
    json = response.json()
    intraday = json.get("Time Series (1min)")
    keys = intraday.keys()
    for key in keys:
        ts = intraday.get(key)
        dp = DataPoint(key)
        open = ts.get("1. open")
        high = ts.get("2. high")
        low = ts.get("3. low")
        close = ts.get("4. close")
        volume = ts.get("5. volume")
        dp.addIntraday(open,high,low,close,volume)
        dictionary[dp.time] = dp

def getTechnicals(dictionary):
    urls = [sma_url, ema_url, vwap_url, macd_url, rsi_url, adx_url]
    technicals = ["SMA","EMA","VWAP","MACD","RSI","ADX"]
    i = 0
    while (i < len(urls)):
        response = requests.get(urls[i])
        json = response.json()
        tech = json.get("Technical Analysis: " + technicals[i])
        if (tech == None):
            print("Empty response, retrying in 10 seconds...")
            time.sleep(10)
        else:
            print("Getting Technical Indicator: " + technicals[i])
            keys = tech.keys()
            for key in keys:
                t = tech.get(key)
                v = t.get(technicals[i])
                if (dictionary.get(key) != None):
                    dictionary.get(key).addTechnical(technicals[i], v)
            i += 1

def writeDictionaryToFile(dictionary):
    filename = "datapoints.json"
    fp = open(filename, "a")
    json_dictionary = json.dumps(dictionary)
    fp.write(json_dictionary)
    print("Wrote results to file: " + filename)

dictionary = {}

getIntraday(dictionary)

getTechnicals(dictionary)

writeDictionaryToFile(dictionary)

Here is the error:

Traceback (most recent call last):
  File "/Users/Jason/Dev/Python/neural-network-example/alphavantage.py", line 124, in <module>
    writeDictionaryToFile(dictionary)
  File "/Users/Jason/Dev/Python/neural-network-example/alphavantage.py", line 113, in writeDictionaryToFile
    json_dictionary = json.dumps(dictionary)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type DataPoint is not JSON serializable

From my understanding, I can use json.dumps() on a common python datatype, string, int, array dictionary, etc. But I cannot use it on custom objects that I've created. I've done research and from my research, I have figured out to use myCustomObject.dict to make the object serializable. How can I use this when I am trying to serialize the entire dictionary?

I'm new to Python, I just can't figure this out. Any help is greatly appreciated.

Answer 1

It's possible to achieve this using a custom JSON serializer, but that may be overkill for your task. A simpler solution is to give your class a couple of methods to convert to JSON and back, via dictionaries.

Your class has quite a lot of fields, so I'll give a simpler example for a toy class, which you should be able to adapt for your purpose:

import json

class Example:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def to_json(self):
        return json.dumps({
            'x': self.x,
            'y': self.y
        })

    @classmethod
    def from_json(cls, s):
        d = json.loads(s)
        return cls(d['x'], d['y'])

Usage:

>>> ex = Example(1, 2)
>>> s = ex.to_json()
>>> s
'{"y": 2, "x": 1}'
>>> ex2 = Example.from_json(s)
>>> ex2.x
1
>>> ex2.y
2

Answer 2

json is a very portable format, but it's also restricted. Only the following things can be serialized with Json:

dicts - {} (all keys must be strings)
lists - []
strings - "string"
integers - 0, 1, 2, ...
True
False
None

So you'll have to transform your object to some combination of these things, and have code to transform it back.

If you are planning on ONLY using python, you may be interested in pickle, which can serialize arbitrary python objects, as long as it can import the modules in which they were defined. Do note that unpacking pickles from unknown sources can lead to remote code executions.