CODEWITHSUNDEEP
javarest
user2430771
user2430771

Reputation: 1422

Read REST api response hosting very large data

I am calling an REST API endpoint which is hosting a very large amount of data. The data quantity is too much that my chrome tab crashes as well (It displays the data for a short time and it's loading even more data causing the tab to crash). Even postman fails to get the data and instead would only return 200 OK code without displaying any response body.

I'm trying to write a java program to consume the response from the API. Is there a way to consume the response without using a lot of memory?

Please let me know if the question is not clear. Thank you !!

Upvotes: 0

Views: 4004

Answers (2)

Alex Chernyshev
Alex Chernyshev

Reputation: 1745

If data is really so large then better to split task:

  1. Download full data via ordinary http client to disk
  2. Make bulk processing , using some streaming approach, similar to SAX parsing for XML: JAVA - Best approach to parse huge (extra large) JSON file

With such split, you will not deal with possible network errors during processing and will keep data consistency.

Upvotes: 1

ralf htp
ralf htp

Reputation: 9422

A possibility is to use a JSON streaming parser like Jackson Streaming API (https://github.com/FasterXML/jackson-docs/wiki/JacksonStreamingApi) for example code see https://javarevisited.blogspot.com/2015/03/parsing-large-json-files-using-jackson.html

For JS there is https://github.com/DonutEspresso/big-json

Upvotes: 2

Related Questions