JSONArray throwing out of memory exception in android

Question

In my app I sync some data at the end of day to the app server.For this I wrap all my data as a JSONArray of JSONObjects.The data mainly includes about 50 pictures each with a size of approx 50kb(along with some text data).All these pictures are encoded using base64 encoding.Everthing works fine when the pictures uploaded(along with some text data) are few in number,but when I upload a large no of pictures ,say around 50 then I see in the logs that all the data is properly formed into the JSONArray,however when I try to display the JSONArray using 'array.toString()' method I encounter an out of memory exception.This I believe is due to the heap getting full(however,when I try making android:largeHeap="true" in the manifest everything is working fine,however I want to avoid using this approach,since this is not a good practice).My intention is just to write this JSONArray value into a file and then break this file into small chunks and send it across to the server. Please guide me of the best approach of writing the JSONAray value to the file which won't lead to OOM issues.Thanks !

Following is the format of the JSONArray:

[{"pid":"000027058451111","popup_time":"2014-01-13 23:36:01","picture":"...base64encoded string......","punching_time":"Absent","status":"Absent"},{"pid":"000027058451111","popup_time":"2014-01-13 23:36:21","picture":"...base64encoded string......","punching_time":"Absent","status":"Absent"}]

Following are the main snippets of my code:

            JSONObject aux;
            JSONArray array = new JSONArray();
            .
            .
            // Looping through each record in the cursor
            for (int i = 0; i < count; i++) {
                aux = new JSONObject();

                try {
                    aux.put("pid", c.getString(c.getColumnIndex("pid")));
                    aux.put("status", c.getString(c.getColumnIndex("status")));
                    aux.put("pop_time", c.getString(c.getColumnIndex("pop_time")));
                    aux.put("punching_time", c.getString(c.getColumnIndex("punching_time")));
                    aux.put("picture", c.getString(c.getColumnIndex("image_str"))); // stores base64encoded picture
                } catch (Exception e) {
                    e.printStackTrace();
                }

                array.put(aux); // Inserting individual objects into the array , works perfectly fine,no error here
                c.moveToNext(); // Moving the cursor to the next record
            }

            Log.d("Log", "length of json array - "+array.length()); // shows me the total no of JSONObjects in the JSONArray,works fine no error

            // HAD GOT OOM HERE
            //Log.d("Log", "JSONArray is - " + array.toString()); 

            if (array.length() != 0){
                try {

                    String responseCode = writeToFile(array);  //Writing the JSONArray value to file,which will then send file to server.

                    if(responseCode.equals("200"))
                        Log.d("Log","Data sent successfully from app to app server");
                    else    
                        Log.d("Log","Data NOT sent successfully from app to app server");
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
            .
            .

            private String writeToFile(JSONArray data) {

            Log.d("Log", "Inside writeToFile");
            File externalStorageDir = new File(Environment.getExternalStorageDirectory().getPath(), "Pictures/File");

            if (!externalStorageDir.exists()) {
                externalStorageDir.mkdirs();
            }

            String responseCode = "";
            File dataFile = new File(externalStorageDir, "File");
    /*      FileWriter writer;
            String responseCode = "";
            try {
                writer = new FileWriter(dataFile);
                writer.append(data);
                writer.flush();
                writer.close();

                responseCode = sendFileToServer(dataFile.getPath(), AppConstants.url_app_server); // Sends the file to server,worked fine for few pictures

            } catch (IOException e) {
                e.printStackTrace();
            }*/


            try {
                FileWriter file = new FileWriter("storage/sdcard0/Pictures/File/File");

                file.write(data.toString());        // GOT OOM here.
                file.flush();
                file.close();
                Log.d("Log","data  written from JSONArray to file");
                responseCode = sendFileToServer(dataFile.getPath(), AppConstants.url_app_server);    // Sends the file to server,worked fine for few pictures
            } catch (IOException e) {
                e.printStackTrace();
            }

            return responseCode;

        }


        public String sendFileToServer(String filename, String targetUrl) {
            .
            .
            // Sends the file to server,worked fine for few pictures
            .
            .
            return response;
        }

Brian Roach · Accepted Answer

Here's the issue. You're trying to load your entire dataset into memory. And you're running out of memory.

Android's JSON classes (and some other JSON libraries) are designed to take a Java object (in memory), serialize it to a parse tree of objects (e.g. JSONObject, JSONArray) (in memory), then convert that tree to a String (in memory) and write it out somewhere.

Specifically in your case (at the moment) it appears what when it converts the parse tree into a String it runs out of memory; That String is effectively doubling the amount of memory required at that point.

To solve your issue you have a few different choices, I'll offer 3:

Don't use JSON at all. Refactor to simply send files and information to your server.
Refactor things so that you only read X images into memory at a time and have multiple output files. Where X is some number of images. Note this is still problematic if your image sizes vary greatly / aren't predictable.
Switch to using Jackson as a JSON library. It supports streaming operations where you can stream the JSON to the output file as you create each object in the array.

Edit to add: for your code, it would look something like this using Jackson:

// Before you get here, have created your `File` object
JsonFactory jsonfactory = new JsonFactory();
JsonGenerator jsonGenerator = 
    jsonfactory.createJsonGenerator(file, JsonEncoding.UTF8);

jsonGenerator.writeStartArray();

// Note: I don't know what `c` is, but if it's a cursor of some sort it
// should have a "hasNext()" or similar you should be using instead of
// this for loop
for (int i = 0; i < count; i++) {

    jsonGenerator.writeStartObject();

    jsonGenerator.writeStringField("pid", c.getString(c.getColumnIndex("pid")));
    jsonGenerator.writeStringField("status", c.getString(c.getColumnIndex("status")));
    jsonGenerator.writeStringField("pop_time", c.getString(c.getColumnIndex("pop_time")));
    jsonGenerator.writeStringField("punching_time", c.getString(c.getColumnIndex("punching_time")));
    // stores base64encoded picture
    jsonGenerator.writeStringField("picture", c.getString(c.getColumnIndex("image_str")));

    jsonGenerator.writeEndObject();

    c.moveToNext(); // Moving the cursor to the next record
}

jsonGenerator.writeEndArray();
jsonGenerator.close();

The above is untested, but it should work (or at least get you going in the right direction).

JSONArray throwing out of memory exception in android

Answers (2)

Related Questions