CODEWITHSUNDEEP
javajson
Dida
Dida

Reputation: 23

How to fix an Out Of memory when importing a large JSON file

I want to import a JSON file with 5 GB size but it shows me an Out of memory error. I set up the JVM by putting -Xmx7700m -Xms7700m -XX: + UseConcMarkSweepGC knowing that I have 8 GB of RAM in my computer but the program execution takes 45 minutes and then shows me this error: i'm using maven depedency "com.googlecode.json-simple" version : 1.1.1

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: Java heap space //if i put -Xmx5000m -Xms5000m

and this is the code of importing of the JSON File

 JSONParser parser = new JSONParser();

    try {
        Object obj = parser.parse(new FileReader("url.json"));

        JSONObject jsonObject =  (JSONObject) obj;
        System.out.println(jsonObject);
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    } catch (ParseException e) {
        e.printStackTrace();
    }

can i find another solution to divide the JSON File into parts and import it part by part?

Upvotes: 2

Views: 3543

Answers (2)

Subarata Talukder
Subarata Talukder

Reputation: 6281

// Create a mapper model class according with your JSON file fields. I gave an example.   
public class JsonToJavaObject {

    @SerializedName("Id")
    @Expose
    private Integer id;

    @SerializedName("Name")
    @Expose
    private String name; 

   // Getter/Setter
}

// Create file from your JSON
File file = new File("url.json"); // Make sure your file name and location

// Make an input stream for the file's data
BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file));
InputStreamReader inputStreamReader = new InputStreamReader(buf, StandardCharsets.UTF_8);

// Read stream of data
try (JsonReader jsonReader = new JsonReader(inputStreamReader)) {
     Gson gson = new GsonBuilder().create();
     // Create JSON object
     JsonToJavaObject = gson.fromJson(jsonReader, JsonToJavaOnline.class);
} catch (Exception e) {
   e.getMessage();
}

Upvotes: 0

riorio
riorio

Reputation: 6816

Assuming your JSON file is a big array of JSON objects, can this assist you?:

import com.google.gson.stream.JsonReader;
import org.json.simple.JSONObject;
...


  JsonReader jsonReader = new JsonReader(new InputStreamReader(new FileInputStream(theFile), StandardCharsets.UTF_8));
  jsonReader.beginArray();
  Gson gson = new GsonBuilder().create();
  while (jsonReader.hasNext()) {
       JSONObject currentJsonObject = gson.fromJson(jsonReader, JSONObject.class);
        // do stuff
   }
   jsonReader.close();

Upvotes: 1

Related Questions