CODEWITHSUNDEEP
pythonjson
Zack Yoshyaro
Zack Yoshyaro

Reputation: 2124

Is it possible to append information to a structure inside of a json file without first reading it into memory?

I'm trying to store a big list of lists in a json file. The lists are generated from a long running process, so I'd like to add the newly generated information into my json file as it becomes available.

Currently, in order to extend the data structure, I'm reading the json into memory as a Python list, appending the new data to that list, and then writing over the old data in the json file with the newly created list. 

def update_json_file(new_data):
    with open('mycoolfile.json', 'rb') as f: 
        jsondata = json.load(f)

    jsondata.append(new_data)
    with open('mycoolfile.json', 'wb') as f: 
        json.dump(jsondata, f)

Is there a better way than reading everything into memory? Surely as the file size grows this will no longer be a viable strategy. Is there a simple way to extend a structure inside of a json file?

Upvotes: 1

Views: 286

Answers (1)

martineau
martineau

Reputation: 123473

Yes, you can, as zaquest said, you can seek to almost the end of the file and overwrite the final ']' of the outer list. Here's something that shows how that might be done:

import json
import os

def append_list(json_filename, new_data):
    with open(json_filename, 'r+b') as f:
        f.seek(-1, os.SEEK_END)
        new_json = json.dumps(new_data)
        f.write(', ' + new_json + ']')

# create a test file
lists = [
    'This is the first list'.split(),
    "and here's another.".split(),
    [10, 2, 4],
]

with open('mycoolfile.json', 'wb') as f:
    json.dump(lists, f)

append_list('mycoolfile.json', 'New data.'.split())

with open('mycoolfile.json', 'rb') as f:
    jsondata = json.load(f)
    print json.dumps(jsondata, indent=4)

Output:

[
    [
        "This",
        "is",
        "the",
        "first",
        "list"
    ],
    [
        "and",
        "here's",
        "another."
    ],
    [
        10,
        2,
        4
    ],
    [
        "New",
        "data."
    ]
]

Upvotes: 1

Related Questions