Optimized way of writing/reading json file

Question

So i have large JSON files (which contain a list of JSON object) and i have to:

• add/remove objects from the file's list of JSON objects
• update object's values (specifically this objects can have list of objects and i might have to add or remove elements from this list)
• read each object in the file (here i would like to load only the file for each object so that i don't have to load the whole file in the RAM keep in mind that this files are very heavy)

Here you can find an example of the file:

[

{
     "id":"1",
     "list":[
         { "id": "1", "example": "example" }, { "id": "2", "example": "example" }
     ]
},
{
     "id":"2",
     "list":[
         { "id": "1", "example": "example" }, { "id": "2", "example": "example" }
     ]
},
{
     "id":"3",
     "list":[
         { "id": "1", "example": "example" }, { "id": "2", "example": "example" }
     ]
},
...
]

For the part of updating the array i have tought of reading the object then change the array and replace the entire object in the file (it may work but i am open to better solutions).

How can i do this? (I am using Java)

Ok so currently i found a way of writing in the JSON file but not appending data to it (this is what i need) here is my code:

public void create(JSONObject document){
        try {
            FileWriter fw = new FileWriter(new File(collectionPath));
            JSONWriter jsonWriter = new JSONWriter(fw);
            jsonWriter.object();
            Iterator<String> keys = document.keys();
            while(keys.hasNext()){
                String key = keys.next();
                jsonWriter.key(key);
                jsonWriter.value(document.get(key));
            }
            jsonWriter.endObject();
            fw.flush();
            fw.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

By doing this i write the file and i need to append the data not to write it (cause this replaces the already written data with the new data and i don't want this).

This is an example of the file keep in mind that this is an example and the real one could weight event gigabytes.

{"uid":"1","password":["keyboardcat","keyboardcat1"],"nestedObject":{"bool":true,"string":"stringa","int":1,"object":{"bool":true,"string":"stringa","int":1}},"username":"test"}

Answer 1

If You use org.json.JSONObject then try this org.json.JSONTokener

for read :

public static JSONObject getJSONFromFile(String path){
        JSONObject result ;
        Path resultPath = Paths.get(path);
        try {
            InputStream stream = Files.newInputStream(resultPath);
            result = new JSONObject(new JSONTokener(stream));
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        return result;
    }

for write :

public static void writeJSON(JSONObject json, String path){
        Path filePath = Paths.get(path);
        if(Files.exists(filePath))
        {
            try(FileWriter writer = new FileWriter(filePath.toFile())){
                writer.write(json.toString(4));
                writer.flush();
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        }
    }

Answer 2

On the jackson-core site, at Usage, simple reading chapter, the Reading and Writing Event Streams example is provided where one can see how to read a large json file efficiently. You'll basically read the array items one by one, change them if you need to and then write them to a new file; if you need to remove some items you'll simply omit writing them while when you have to insert new items you'll simply write them after reading&writing (aka copying) the appropriate number of items from the original file. In the end you can replace the original file with the new one.

Below is an example of writing a large array of objects (HashMap here but could be any other) with writePOJOField method. I use jackson-core and jackson-databind (see https://github.com/FasterXML/jackson).

// given
ByteArrayOutputStream stream = new ByteArrayOutputStream();
JsonFactory jfactory = new JsonFactory(new ObjectMapper());
JsonGenerator jGenerator = jfactory.createGenerator(stream, JsonEncoding.UTF8);

// when
jGenerator.writeStartObject();
jGenerator.writeStringField("name", "Tom");
jGenerator.writeNumberField("age", 25);

jGenerator.writeFieldName("objects");
jGenerator.writeStartArray();
IntStream.range(0, 3).forEach(it -> {
    try {
        jGenerator.writePOJO(new HashMap<String, String>() {{
            put("key1", "value1");
            put("key2", "value2");
        }});
    } catch (IOException e) {
        e.printStackTrace();
        throw new RuntimeException(e);
    }
});
jGenerator.writeEndArray();

jGenerator.writeFieldName("strings");
jGenerator.writeStartArray();
jGenerator.writeString("textX");
jGenerator.writeString("textY");
IntStream.range(0, 3).forEach(it -> {
    try {
        jGenerator.writeString("text-" + it);
    } catch (IOException e) {
        e.printStackTrace();
        throw new RuntimeException(e);
    }
});
jGenerator.writeEndArray();

jGenerator.writeNullField("nullField");
jGenerator.writeEndObject();
jGenerator.close();