CODEWITHSUNDEEP
restheap-memorylarge-data
Freya Ren
Freya Ren

Reputation: 2164

How can REST API pass large JSON?

I am building a REST API and facing this issue: How can REST API pass very large JSON?

Basically, I want to connect to Database and return the training data. The problem is in Database I have 400,000 data. If I wrap them into a JSON file and pass through GET method, the server would throw Heap overflow exception.

What methods we can use to solve this problem?

DBTraining trainingdata = new DBTraining();
@GET
@Produces("application/json")
@Path("/{cat_id}")
public Response getAllDataById(@PathParam("cat_id") String cat_id) {
    List<TrainingData> list = new ArrayList<TrainingData>();
    try {
        list = trainingdata.getAllDataById(cat_id);
        Gson gson = new Gson();
        Type dataListType = new TypeToken<List<TrainingData>>() {
        }.getType();
        String jsonString = gson.toJson(list, dataListType);
        return Response.ok().entity(jsonString).header("Access-Control-Allow-Origin", "*").header("Access-Control-Allow-Methods", "GET").build();

    } catch (SQLException e) {
        logger.warn(e.getMessage());
    }
    return null;
}

Upvotes: 8

Views: 18348

Answers (3)

Darrel Miller
Darrel Miller

Reputation: 142252

HTTP has the notion of chunked encoding that allows you send a HTTP response body in smaller pieces to prevent the server from having to hold the entire response in memory. You need to find out how your server framework supports chunked encoding.

Upvotes: 0

Rodrigo Veron
Rodrigo Veron

Reputation: 66

My suggestion is no to pass the data as a JSON but as a file using multipart/form-data. In your file, each line could be a JSON representing a data record. Then, it would be easy to use a FileOutputStream to receive te file. Then, you can process the file line by line to avoid memory problems.

A Grails example:

    if(params.myFile){
           if(params.myFile instanceof org.springframework.web.multipart.commons.CommonsMultipartFile){
           def fileName = "/tmp/myReceivedFile.txt"
           new FileOutputStream(fileName).leftShift(params.myFile.getInputStream())
       }
       else
           //print or signal error
    }

You can use curl to pass your file:

    curl -F "myFile=@/mySendigFile.txt" http://acme.com/my-service

More details on a similar solution on https://stackoverflow.com/a/13076550/2476435

Upvotes: 0

Carlos
Carlos

Reputation: 1176

The RESTful way of doing this is to create a paginated API. First, add query parameters to set page size, page number, and maximum number of items per page. Use sensible defaults if any of these are not provided or unrealistic values are provided. Second, modify the database query to retrieve only a subset of the data. Convert that to JSON and use that as the payload of your response. Finally, in following HATEOAS principles, provide links to the next page (provided you're not on the last page) and previous page (provided you're not on the first page). For bonus points, provide links to the first page and last page as well.

By designing your endpoint this way, you get very consistent performance characteristics and can handle data sets that continue to grow.

The GitHub API provides a good example of this.

Upvotes: 4

Related Questions