CODEWITHSUNDEEP
jsondjangodatabasepostgresqldjango-rest-framework
Kabiljan Tanaguzov
Kabiljan Tanaguzov

Reputation: 323

save large json objects in postgres, Django

I have a model in db:

class Test(models.Model):
    title = models.CharField(max_length=32, verbose_name='title', default='')
    json = models.JSONField(default=dict)
    ...

I get the data from the front and save it to the db, requests come in quite often. The average weight of a json field is 10MB, but it can vary greatly and I don’t understand how it would be better for me to save and give it, I don’t do any work with json on server. To begin with, I think need to compress this json and save it to the database and, when requested to receive it, decompress it. Can you please advise me on the best way to save memory and query execution time.

Also, is it worth removing this json in a separate table so that changing other data in the test table takes less time, or is it better to use Test.objects.update()?

Upvotes: 2

Views: 1899

Answers (1)

Swift
Swift

Reputation: 1711

Whilst I appreciate its not going to be the simplest solution and certainly not the quickest or easiest, however, storing "large json blobs" in the database is probably never a good idea.

If you have a large object and need to represent that data in a django model, then I would suggest you do exactly that.

Rather than attempting to save the whole json blob as one string and then loading it from the database and subsequently into memory every time you want to read one attribute? There's huge performance implications to this that you would find difficult to mitigate without some form of caching.

If you REALLY MUST store the data as the whole json blob, then it would probably be better to store it on the file system itself and then create a model which has a FileField instead of a JSONField

You could also do what would be a much more, Pythonic / Django-esque, which would be to dissect your object, each of its attributes, and its values, and then design your models so that it can store each of the data types. As a simple example:

{
    'field_a': 'string_value',
    'm2m_field_b': [
        {
            'field_c': 6.4,
            'field_d': 'other string values'
        }
    ]
    ...
 }

You have here, 2 or potentially more, depending on the complexity of the json blob.

  1. ParentModelA 1a) field_a is a string field
  2. ChildModelA 2a) fk field to parent object 2b) field_c is a float (preferably, Decimal) field and is on our related model 2c) field_d is another string field and is again on our related model.

I know, it sounds really complicated at first, but once you start getting the hang of modelling these types of objects, you realise that its very repetitious and you won't need to write that much extra code.

Why would you go this far you ask? It seems like a lot of work without much benefit? To the untrained eye, yes, no offense. What you have to take into consideration is the fact that you can't use django's useful interactions with the database, like aggregation, or prefetch and select related, so you lose a lot of the advantages and leverage a the django database API has to offer.

Also take for consideration if you wanted to load one of the field values on a simple page to display the data to a user in a User Interface.

With your model, you have to load the entire json blob, then display the simple string and decimal value.

If you had a set of models, you can traverse djangos related fields and even use the objects directly in a template using the usual dot notation.

Upvotes: 2

Related Questions