CODEWITHSUNDEEP
pythonsqldjangodjango-modelsnosql
Andrew
Andrew

Reputation: 3999

Hybrid sql/nosql data store in Django for document-based fields?

I am creating an application that needs to allow users to create custom fields, which I think would be best stored in a document-based (basically a serialized dictionary) model field.

I am concerned that I would run into performance issues storing these potentially very large documents in a SQL database, so I thought instead of storing the documents in the SQL database, I would just store pointers in the SQL database to the documents. The documents themselves would then be stored in a separate NoSQL database.

Assuming this structure makes sense, what is the best way to go about constructing a field that stores custom data fields in this manner? Optimally, these custom fields would be accessible on the object as attributes and would be denoted as custom with a "_c" appended to the name. E.g. created_date would become created_date_c on the django model object. I'm thinking custom managers would be best to tackle this: https://docs.djangoproject.com/en/dev/topics/db/managers/

EDIT: My SQL database is MySQL. Also, documents could make more sense as columns (as e.g. in Cassandra). Thoughts on this would be helpful.

Upvotes: 0

Views: 682

Answers (1)

alko
alko

Reputation: 48317

As far as I know, there is no best approach to this task, but I advise you to look into two other options, that has its own pros and contras:

  • Store your user defined data in a JSONfield or pickled field. This will save you a lot of efforts in writing custom managers and NoSQL. If you are worried about storing it along with your fixed structure data, store it in a separate model with one-to-one relationship in a separate InnoDB file for example.

  • Store yours user data in a generalized (field_id, field_name, content_type) and (object_id, field_id, field_value) dictionaries. They can be split by field types (i.e. int, string, float etc.). This approach won't give you well performing data model from scratch, but smart indexing and partitioning can make it worth noting. And your data query, model structure enforcement will be alot easier from other approaches.

  • If you consider using NoSQL or other db for your mutable content, be sure to choose one that has means for your data efficient querying, see this discussion and wiki.

Upvotes: 1

Related Questions