CODEWITHSUNDEEP
javagson
snaggs
snaggs

Reputation: 5713

java.lang.OutOfMemoryError: Java heap space when try to convert Java Object to Json String

I try to convert csv file to Json file 200K of objects where object represents 1 row in csv.

I have Java installed on 32 bit and Project configuration VM arguments: -Xmx1024m

However I get:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
    at java.lang.AbstractStringBuilder.append(Unknown Source)
    at java.lang.StringBuffer.append(Unknown Source)
    at java.io.StringWriter.write(Unknown Source)
    at com.google.gson.stream.JsonWriter.string(JsonWriter.java:478)
    at com.google.gson.stream.JsonWriter.value(JsonWriter.java:328)
    at com.google.gson.Streams.write(Streams.java:113)
    at com.google.gson.Streams.write(Streams.java:136)
    at com.google.gson.Streams.write(Streams.java:136)
    at com.google.gson.Streams.write(Streams.java:124)
    at com.google.gson.Streams.write(Streams.java:136)
    at com.google.gson.Gson.toJson(Gson.java:362)
    at com.google.gson.Gson.toJson(Gson.java:346)
    at com.google.gson.Gson.toJson(Gson.java:260)
    at com.google.gson.Gson.toJson(Gson.java:240)
    at ConvertFromCsv2JsonTWC.init(ConvertFromCsv2JsonTWC.java:186)
    at ConvertFromCsv2JsonTWC.main(ConvertFromCsv2JsonTWC.java:48)

In row:

Gson gson = new Gson();

String output = gson.toJson(container);// <---- crash

for 50k rows it works fine.

This is a template of Json I build:

{
    "crs": {
        "type": "none"
    },
    "type": "FeatureCollection",
    "features": [{
        "geometry": {
            "type": "Point"
        },
        "properties": {
            "ap mac": "00:11:22:33:44:55",
            "ssid": "WiFi",
            "lat": "35.111111",
            "long": "-118.11111",
            "address": "370 xxxxxx",
            "city": "xxxxxxx",
            "state": "CA",
            "zip code": "11111",
            "country": "US",
            "business n": "",
            "location c": "Health Club/Gym",
            "location q": "",
            "indoor fla": "yes"
        },
        "point": [35.390284,
        -118.9929],
        "id": 0,
        "type": "Feature"
    },
          {...},
          ... 
          200000...

So I have 200K objects in properties list

The workaround is to create separate files 20k per each but it not good way.

How can I solve this?

Thanks,

Upvotes: 8

Views: 9575

Answers (4)

Egemen Hamut&#231;u
Egemen Hamut&#231;u

Reputation: 1712

This is how you stream data into Json as String using Gson Streaming API

@Nullable
public static String streamContainersIntoJsonString(List<Container> containers) {
    try {
        Gson gson = new Gson();
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        JsonWriter writer = new JsonWriter(new OutputStreamWriter(out, "UTF-8"));
        writer.setIndent("  ");
        writer.beginArray();
        for (Container container : containers) {
            gson.toJson(container, Container.class, writer);
        }
        writer.endArray();
        writer.close();

        return out.toString("UTF-8");
    } catch (IOException e) {
        e.printStackTrace();
    }

    return null;
}

Upvotes: 1

Elliott Frisch
Elliott Frisch

Reputation: 201457

I suggest you use streaming instead of trying to copy it all into a String.

Upvotes: 4

sethcall
sethcall

Reputation: 2907

You can use the Gson streaming API to instead walk (stream) the data, instead of trying to load it all at once.

Upvotes: 4

Steve P.
Steve P.

Reputation: 14699

There is too much data to read all of it in at once and keep it in memory. You should break it up into smaller parts and process it piece-wise.

Upvotes: 0

Related Questions