CODEWITHSUNDEEP
javamultithreadingspring-bootrestdownload
Luke
Luke

Reputation: 1253

Spring boot app and what approach to use to download bulk data

I have spring boot application with basic REST API. My question is what shall we use to download some bulk data? What is preferable way how to download bulk data without memory leak? Let's suppose we have 10 million records.

Here are some approaches but not sure:

  1. download with PipedInputStream when data are written with PipedOutputStream in separated thread. Is it fine or it is not good choice?
  2. download with ByteArrayOutputStream when data are written into temp file in separated thread and after finish it is ready to download. We can mark this operation with some flags for end user eg. DOWNLOAD_ACTIVE, DOWNLOAD_DONE. The user is just initiating download with result flag DOWNLOAD_ACTIVE and trying to ping server for response flag DOWNLOAD_DONE. When it is done then the user is going to send request to download data.

Summary 2)

1. initiate request to download data - ACTIVE state
2. ping server and server returns current state - ACTIVE or DONE
3. if final state is DONE then user initiate final request to download data

Thanks

Upvotes: 0

Views: 250

Answers (1)

Dhaval Gajjar
Dhaval Gajjar

Reputation: 2925

You can use the second approach. Which can prepare data in the background and once it's ready you can download it.

  1. Send a request to prepare data. The server responds with a UUID.
  2. Server starts preparing files in the background. The server has a Map that has the key with a new UUID and value as status ACTIVE.
  3. Client saved UUID and checks the server after a certain interval by passing the UUID.
  4. Once the server finishes the task it will update the Map for the given UUID value as status DONE.
  5. As the status is DONE next request will provide the status DONE and UI and send another request to download the file.

The above approach will only work if you don't refresh the page. As page refresh will clear the UUID and you have to proceed again.

To achieve this after refresh/cross-logins then you need to use a database table instead of Map. Store the username along with other information and inform the user once it's ready.

Upvotes: 1

Related Questions