Android : java.lang.OutOfMemoryError: Failed to allocate with free bytes and 70MB until OOM when using gson.toJson()

Question

In my Android application I am getting below exception when I try to sync my data to the server which is large in size. I get this exception when data size is more than 20 MB I think. I am saving bitmap images as string using base64 encoding after sacle down the image size which makes such huge data.

04-18 13:51:51.957  16199-16816/com.example.myproject.app E/art﹕ Throwing OutOfMemoryError "Failed to allocate a 128887990 byte allocation with 16777216 free bytes and 70MB until OOM"
04-18 13:51:52.037  16199-16816/com.example.myproject.app E/AndroidRuntime﹕ FATAL EXCEPTION: Thread-4482
Process: com.example.myproject.app, PID: 16199
java.lang.OutOfMemoryError: Failed to allocate a 128887990 byte allocation with 16777216 free bytes and 70MB until OOM
    at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:95)
    at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:146)
    at java.lang.StringBuffer.append(StringBuffer.java:219)
    at java.io.StringWriter.write(StringWriter.java:167)
    at com.google.gson.stream.JsonWriter.string(JsonWriter.java:570)
    at com.google.gson.stream.JsonWriter.value(JsonWriter.java:419)
    at com.google.gson.internal.bind.TypeAdapters$16.write(TypeAdapters.java:426)
    at com.google.gson.internal.bind.TypeAdapters$16.write(TypeAdapters.java:410)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:112)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:239)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:97)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:61)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)

How to solve this problem? I know that this comes when I convert the data from class to json using Gson. Below is my code:

SimpleDateFormat dtf=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss",Locale.ENGLISH);
GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(Date.class, new JsonDeserializer<Date>() {

        @Override
        public Date deserialize(JsonElement json, Type type, JsonDeserializationContext deserializationContext) throws JsonParseException {
            String frStr = json.getAsJsonPrimitive().getAsString();
            Date retDate =null;
            try {
                retDate = dtf.parse(frStr);
            } catch (ParseException e) {
                e.printStackTrace();
            }
            return retDate;
        }
    });
    builder.registerTypeAdapter(Date.class, new JsonSerializer<Date>() {
            @Override
            public JsonElement serialize(Date src, Type typeOfSrc, JsonSerializationContext context) {
                String jsDate = dtf.format(src);
                return new JsonPrimitive(jsDate);
            }
    });
    builder.registerTypeAdapter(byte[].class, new JsonDeserializer<byte[]>() {
            @Override
            public byte[] deserialize(JsonElement json, Type type, JsonDeserializationContext deserializationContext) throws JsonParseException {
                return Base64.decode(json.getAsString(), Base64.NO_WRAP);
            }
    });
    gson = builder.create();


    attDataAcc.setAttList(attList);
    String jsonAttAccts = gson.toJson(attDataAcc, AttachmentDataList.class);
        HttpEntity<String> entityAtt = new HttpEntity<String>(jsonAttAccts,headers);
        ResponseEntity<String> restResA = restTemplate.exchange(strUrl+"/saveAttToServer", HttpMethod.POST, entityAtt, String.class);

public class Attachment implements Serializable {

            @DatabaseField(columnName = "id",id = true)
            private String id;

            @DatabaseField(columnName = "user_id")
            private Integer userId;

            @DatabaseField(columnName = "attachment_id")
            private String attachmentId;

            @DatabaseField(columnName = "file_name")
            private String fileName;

            @DatabaseField(columnName = "file_data")
            private String fileData;

            @DatabaseField(columnName = "date",dataType=DataType.DATE)
            private Date date;

            public Attachment() {
                super();
                // TODO Auto-generated constructor stub
            }

            public Attachment(String id, Integer userId, String attachmentId, String fileName, String fileData, Date date) {
                this.id = id;
                this.userId = userId;
                this.attachmentId = attachmentId;
                this.fileName = fileName;
                this.fileData = fileData;
                this.date = date;
            }

            public String getId() {
                return id;
            }

            public void setId(String id) {
                this.id = id;
            }

            public Integer getUserId() {
                return userId;
            }

            public void setUserId(Integer userId) {
                this.userId = userId;
            }

            public String getAttachmentId() {
                return attachmentId;
            }

            public void setAttachmentId(String attachmentId) {
                this.attachmentId = attachmentId;
            }

            public String getFileName() {
                return fileName;
            }

            public void setFileName(String fileName) {
                this.fileName = fileName;
            }

            public String getFileData() {
                return fileData;
            }

            public void setFileData(String fileData) {
                this.fileData = fileData;
            }

            public Date getDate() {
                return date;
            }

            public void setDate(Date date) {
                this.date = date;
            }

            @Override
            public boolean equals(Object o) {
                if (this == o) return true;
                if (o == null || getClass() != o.getClass()) return false;

                Attachment that = (Attachment) o;

                if (id != null ? !id.equals(that.id) : that.id != null) return false;
                if (userId != null ? !userId.equals(that.userId) : that.userId != null) return false;
                if (attachmentId != null ? !attachmentId.equals(that.attachmentId) : that.attachmentId != null) return false;
                if (fileName != null ? !fileName.equals(that.fileName) : that.fileName != null) return false;
                if (fileData != null ? !fileData.equals(that.fileData) : that.fileData != null) return false;
                if (date != null ? !date.equals(that.date) : that.date != null) return false;

            }

            @Override
            public int hashCode() {
                int result = id != null ? id.hashCode() : 0;
                result = 31 * result + (userId != null ? userId.hashCode() : 0);
                result = 31 * result + (attachmentId != null ? attachmentId.hashCode() : 0);
                result = 31 * result + (fileName != null ? fileName.hashCode() : 0);
                result = 31 * result + (fileData != null ? fileData.hashCode() : 0);
                result = 31 * result + (date != null ? date.hashCode() : 0);
                return result;
            }

            @Override
            public String toString() {
                return userFileName;
            }

        }

        public class AttachmentDataList implements Serializable {
            private ArrayList<Attachment> attList;

            public ArrayList<Attachment> getAttList() {
                return attList;
            }

            public void setAttList(ArrayList<Attachment> attList) {
                this.attList = attList;
            }
        }

Answer 1

You're running into OutOfMemoryError because you're using inefficient and very memory-consuming Base64 transformations. Another hit here is Gson: it does not provide any raw-write methods for JsonWriter and JsonReader classes: the most you can do here is writing/reading a single string value. Collecting a huge input to a single string is another very memory-consuming operation: check your stacktrace to make sure that a string builder instance is used under the very hood -- and this is just to write a single value to the output stream. In short, it all looks like this (if I'm not mistaken about your code, because it seems to lack really important parts, so I'm just trying to reconstruct your scenario):

• Obtaining a byte array (that would be a new object, probably a clone of another byte array);
• Converting the byte array to a Base64 encoded string (it also hits performance because the would clone the byte array to create a defensive copy);
• Converting ALL to a String gson.toJson(attDataAcc, AttachmentDataList.class); -- another huge hit.

All of this is extremely memory consuming. And it would be nice if Gson could support raw write to the output stream, but currently it lacks any of it.

In theory, you might overcome this issue by just writing to the underlying streams (probably directly from your byte array source without any massive transformations because Base64 can be streamed as well thus consuming memory minimum). You mentioned Gson 2.6.2, but I'm working with Gson 2.8.0, so the below solution can 100% work with Gson 2.8.0 only, and may not work even for any other minor Gson versions because it uses reflection in order to "hack" the JsonWriter class.

final class ByteArrayTypeAdapter
        extends TypeAdapter<byte[]> {

    // These two methods and one field from the super class privates are necessary to make it all work  
    private static final Method writeDeferredNameMethod;
    private static final Method beforeValueMethod;
    private static final Field writerField;

    static {
        try {
            writeDeferredNameMethod = JsonWriter.class.getDeclaredMethod("writeDeferredName");
            writeDeferredNameMethod.setAccessible(true);
            beforeValueMethod = JsonWriter.class.getDeclaredMethod("beforeValue");
            beforeValueMethod.setAccessible(true);
            writerField = JsonWriter.class.getDeclaredField("out");
            writerField.setAccessible(true);
        } catch ( final NoSuchMethodException | NoSuchFieldException ex ) {
            throw new RuntimeException(ex);
        }
    }

    // This type adapter is effectively a singleton having no any internal state
    private static final TypeAdapter<byte[]> byteArrayTypeAdapter = new ByteArrayTypeAdapter();

    private ByteArrayTypeAdapter() {
    }

    // But making the constructor private and providing access to the instance via the method, we make sure that the only instance exists and it's safe
    static TypeAdapter<byte[]> getByteArrayTypeAdapter() {
        return byteArrayTypeAdapter;
    }

    @Override
    public void write(final JsonWriter out, final byte[] bytes)
            throws IOException {
        try {
            // Since we're writing a byte[] array, that's probably a field value, make sure that the corresponding property name has been written to the output stream
            writeDeferredNameAndFlush(out);
            // Now simulate JsonWriter.value(byte[]) if such a method could exist
            writeRawBase64ValueAndFlush(bytes, (Writer) writerField.get(out));
        } catch ( IllegalAccessException | InvocationTargetException ex ) {
            throw new IOException(ex);
        }
    }

    @Override
    public byte[] read(final JsonReader in) {
        // If necessary, requires more hacks...
        // And this is crucial for the server-side:
        // In theory, the client can generate HUGE Base64 strings,
        // So the server could crash with OutOfMemoryError too
        throw new UnsupportedOperationException();
    }

    private static void writeDeferredNameAndFlush(final Flushable out)
            throws IOException, IllegalAccessException, InvocationTargetException {
        writeDeferredNameMethod.invoke(out);
        beforeValueMethod.invoke(out);
        // Flush is necessary: the JsonWriter does not know that we're using its private field intruding to its privates and may not flush
        out.flush();
    }

    private static void writeRawBase64ValueAndFlush(final byte[] bytes, final Writer writer)
            throws IOException {
        // Writing leading "
        writer.write('\"');
        // This comes from Google Guava
        final BaseEncoding baseEncoding = BaseEncoding.base64();
        final OutputStream outputStream = baseEncoding.encodingStream(writer);
        // This too
        // Note that we just r_e_d_i_r_e_c_t streams on fly not making heavy transformations
        ByteStreams.copy(new ByteArrayInputStream(bytes), outputStream);
        // This is necessary too
        outputStream.close();
        // Writing trailing "
        writer.write('\"');
        // Flush again to keep it all in sync
        writer.flush();
    }

}

I know it's a hack, but it's better than just getting OutOfMemoryError constantly.

Now, just make it work with Spring RestTemplates:

// Gson is thread-safe and can be re-used
private static final Gson gson = new GsonBuilder()
        // SimpleDateFormat may be NOT thread-safe so you should not share the single SimpleDateFormat between threads
        // However Gson supports date/time formats out of box
        .setDateFormat("yyyy-MM-dd HH:mm:ss")
        // Registering byte[] to the type adapter
        .registerTypeAdapter(byte[].class, getByteArrayTypeAdapter())
        .create();

private static final RestTemplate restTemplate = new RestTemplate();
private static final String URL = "http://localhost";

public static void main(final String... args) {
    sendPostRequest("hello world".getBytes(), byte[].class);
}

private static void sendPostRequest(final Object object, final Type type) {
    // This is where we're binding the output stream I was asking in the question comments
    final RequestCallback requestCallback = request -> gson.toJson(object, type, new OutputStreamWriter(request.getBody()));
    // Spring RestTemplates stuff here...
    final SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
    requestFactory.setBufferRequestBody(false);
    final ResponseExtractor<String> responseExtractor = new HttpMessageConverterExtractor<>(String.class, restTemplate.getMessageConverters());
    restTemplate.setRequestFactory(requestFactory);
    // Let it fly
    restTemplate.execute(URL, POST, requestCallback, responseExtractor);
}

Note that you might write a specialized type adapter for special types that could write directly to output stream so you could not get rid of byte[] at all. You could also vote up for this issue at the official Gson issue tracker: https://github.com/google/gson/issues/971 and probably there could no any need in using any Java Reflection API hacks in a future version of Gson.