Mutable JSON object type not tracking changes SQLAlchemy (sqlalchemy-json)

Question

Ok: I'm building a Flask app with Flask-SQLAlchemy and PostgreSQL and I'd like to make a mutable JSON type column in my database. There are many examples online (and many questions here about) custom mutable object types in SQLAlchemy. I found this one: sqlalchemy-json (with a complete writeup by the author here) which deals with mutable JSON object types. In theory it presents JsonObject, a JSON object type with change tracking for dict and list at base level, and NestedJsonObject, a JSON object type with nested change tracking for dict and list. It looks sweet.

I cannot for the life of me get it to work, though I have every confidence it does. As a note: the above author writeup does not show an example of implementing the object type in a column, so since I'm a newb I'm in all probability getting this next part wrong. I have however consulted the sqlalchemy.ext.mutable reference and it seems right.

Here is my models.py:

from application import app
from flask_sqlalchemy import SQLAlchemy
from flask_login import UserMixin
import sqlalchemy_json # I'm using Alembic for migrations and make this import in my script.py.mako, too, in case that matters
from sqlalchemy_json import NestedJsonObject

db = SQLAlchemy(app)

class User(UserMixin, db.Model): #flask-sqlalchemy provides a Base declaration with db.Model
    __tablename__ = 'users'

    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String(80), unique=True)
    data = db.Column(NestedJsonObject)

    def __init__(self, name, data):
        self.name = name
        self.data = data

    def __repr__(self):
        return "" % (self.id, self.name, self.data)

...

Given that, this is an example of working with my User:

data = {}
data['a']= {'b': 'c', 'd': 'e'}
user = User( ... , data)
db.session.add(user)
db.session.commit()
#

That initial commit works. But subsequent commits do not:

user = db.session.query(User).filter( ... ).first()
user.data['foo']={}
db.session.commit()
#
user.data['foo']['bar'] = {'x': 'x', 'z': 'z'}
db.session.commit()
#

You get the point. For clarity, here are the two files in the module with the author's notations:

alchemy.py

Please note the .associate_with's at the bottom of the file:

# Third-party modules
try:
  import simplejson as json
except ImportError:
  import json

import sqlalchemy
from sqlalchemy.ext import mutable

# Custom modules
from . import track


class NestedMutable(mutable.MutableDict, track.TrackedDict):
  """SQLAlchemy `mutable` extension dictionary with nested change tracking."""
  def __setitem__(self, key, value):
    """Ensure that items set are converted to change-tracking types."""
    super(NestedMutable, self).__setitem__(key, self.convert(value, self))

  @classmethod
  def coerce(cls, key, value):
    """Convert plain dictionary to NestedMutable."""
    if isinstance(value, cls):
      return value
    if isinstance(value, dict):
      return cls(value)
    return super(cls).coerce(key, value)


class _JsonTypeDecorator(sqlalchemy.TypeDecorator):
  """Enables JSON storage by encoding and decoding on the fly."""
  impl = sqlalchemy.String

  def process_bind_param(self, value, dialect):
    return json.dumps(value)

  def process_result_value(self, value, dialect):
    return json.loads(value)


class JsonObject(_JsonTypeDecorator):
  """JSON object type for SQLAlchemy with change tracking as base level."""


class NestedJsonObject(_JsonTypeDecorator):
  """JSON object type for SQLAlchemy with nested change tracking."""


mutable.MutableDict.associate_with(JsonObject)
NestedMutable.associate_with(NestedJsonObject)

track.py

#!/usr/bin/python
"""This module contains the tracked object classes.
TrackedObject forms the basis for both the TrackedDict and the TrackedList.
A function for automatic conversion of dicts and lists to their tracked
counterparts is also included.
"""

# Standard modules
import itertools
import logging


class TrackedObject(object):
  """A base class for delegated change-tracking."""
  _type_mapping = {}

  def __init__(self, *args, **kwds):
    self.logger = logging.getLogger(type(self).__name__)
    self.logger.debug('%s: __init__', self._repr())
    self.parent = None
    super(TrackedObject, self).__init__(*args, **kwds)

  def changed(self, message=None, *args):
    """Marks the object as changed.
    If a `parent` attribute is set, the `changed()` method on the parent will
    be called, propagating the change notification up the chain.
    The message (if provided) will be debug logged.
    """
    if message is not None:
      self.logger.debug('%s: %s', self._repr(), message % args)
    self.logger.debug('%s: changed', self._repr())
    if self.parent is not None:
      self.parent.changed()

  @classmethod
  def register(cls, origin_type):
    """Registers the class decorated with this method as a mutation tracker.
    The provided `origin_type` is mapped to the decorated class such that
    future calls to `convert()` will convert the object of `origin_type` to an
    instance of the decorated class.
    """
    def decorator(tracked_type):
      """Adds the decorated class to the `_type_mapping` dictionary."""
      cls._type_mapping[origin_type] = tracked_type
      return tracked_type
    return decorator

  @classmethod
  def convert(cls, obj, parent):
    """Converts objects to registered tracked types
    This checks the type of the given object against the registered tracked
    types. When a match is found, the given object will be converted to the
    tracked type, its parent set to the provided parent, and returned.
    If its type does not occur in the registered types mapping, the object
    is returned unchanged.
    """
    obj_type = type(obj)
    for origin_type, replacement in cls._type_mapping.iteritems():
      if obj_type is origin_type:
        new = replacement(obj)
        new.parent = parent
        return new
    return obj

  @classmethod
  def convert_iterable(cls, iterable, parent):
    """Returns a generator that performs `convert` on every of its members."""
    return (cls.convert(item, parent) for item in iterable)

  @classmethod
  def convert_iteritems(cls, iteritems, parent):
    """Returns a generator like `convert_iterable` for 2-tuple iterators."""
    return ((key, cls.convert(value, parent)) for key, value in iteritems)

  @classmethod
  def convert_mapping(cls, mapping, parent):
    """Convenience method to track either a dict or a 2-tuple iterator."""
    if isinstance(mapping, dict):
      return cls.convert_iteritems(mapping.iteritems(), parent)
    return cls.convert_iteritems(mapping, parent)

  def _repr(self):
    """Simple object representation."""
    return '<%(namespace)s.%(type)s object at 0x%(address)0xd>' % {
        'namespace': __name__,
        'type': type(self).__name__,
        'address': id(self)}


@TrackedObject.register(dict)
class TrackedDict(TrackedObject, dict):
  """A TrackedObject implementation of the basic dictionary."""
  def __init__(self, source=(), **kwds):
    super(TrackedDict, self).__init__(itertools.chain(
        self.convert_mapping(source, self),
        self.convert_mapping(kwds, self)))

  def __setitem__(self, key, value):
    self.changed('__setitem__: %r=%r', key, value)
    super(TrackedDict, self).__setitem__(key, self.convert(value, self))

  def __delitem__(self, key):
    self.changed('__delitem__: %r', key)
    super(TrackedDict, self).__delitem__(key)

  def clear(self):
    self.changed('clear')
    super(TrackedDict, self).clear()

  def pop(self, *key_and_default):
    self.changed('pop: %r', key_and_default)
    return super(TrackedDict, self).pop(*key_and_default)

  def popitem(self):
    self.changed('popitem')
    return super(TrackedDict, self).popitem()

  def update(self, source=(), **kwds):
    self.changed('update(%r, %r)', source, kwds)
    super(TrackedDict, self).update(itertools.chain(
        self.convert_mapping(source, self),
        self.convert_mapping(kwds, self)))


@TrackedObject.register(list)
class TrackedList(TrackedObject, list):
  """A TrackedObject implementation of the basic list."""
  def __init__(self, iterable=()):
    super(TrackedList, self).__init__(self.convert_iterable(iterable, self))

  def __setitem__(self, key, value):
    self.changed('__setitem__: %r=%r', key, value)
    super(TrackedList, self).__setitem__(key, self.convert(value, self))

  def __delitem__(self, key):
    self.changed('__delitem__: %r', key)
    super(TrackedList, self).__delitem__(key)

  def append(self, item):
    self.changed('append: %r', item)
    super(TrackedList, self).append(self.convert(item, self))

  def extend(self, iterable):
    self.changed('extend: %r', iterable)
    super(TrackedList, self).extend(self.convert_iterable(iterable, self))

  def remove(self, value):
    self.changed('remove: %r', value)
    return super(TrackedList, self).remove(value)

  def pop(self, index):
    self.changed('pop: %d', index)
    return super(TrackedList, self).pop(index)

  def sort(self, cmp=None, key=None, reverse=False):
    self.changed('sort')
    super(TrackedList, self).sort(cmp=cmp, key=key, reverse=reverse)

Also,

init.py

from .alchemy import NestedJsonObject, NestedMutable, JsonObject

__all__ = (
    'NestedJsonObject',
    'NestedMutable',
    'JsonObject'
)

Thank you for reading this far. If you have any suggestions, please let me know. If this is a duplicate (I couldn't quite find one myself but they could exist), please mark this as so. And also, if you yourself find the above code a good addition to your project, please support the module's author. I am not he/she. Here's his/her LICENSE:

Copyright (c) 2014, Elmer de Looff 
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Mutable JSON object type not tracking changes SQLAlchemy (sqlalchemy-json)

alchemy.py

track.py

init.py

Answers (1)

Related Questions