CODEWITHSUNDEEP
java
ReactivePsycho
ReactivePsycho

Reputation: 19

Returning a very large data from a rest api

I have a huge amount of data (~4 GB) returned from db as an iterable list. I want to return this data from my rest api as a JSON. I can only use 512 mb of Java heap for this task. What approach I can take for this? I have tried Java stream but as of now, I always hit heap out of space error. Any example will be of great help. ☺️

Upvotes: 0

Views: 4827

Answers (1)

S. Miranda
S. Miranda

Reputation: 138

I propose this solution :

  1. Add a LIMIT on your query, to get only a part of your data (< 512mb)
  2. Process the partial data you got and return it to the client with a flag "uncomplete".
  3. Ask the server the following part
  4. Repeat step 1 to 3 until there's no data to process
  5. Return the last part with the flag "complete"
  6. Profit

Upvotes: 2

Related Questions