CODEWITHSUNDEEP
javaspringrestfile-upload
d123
d123

Reputation: 437

Download and read on a large csv file in java Spring Rest service

I have a Spring rest service which runs periodically after every 10 min. Service knows the path to a file stored on remote server. Size of the file is around 2GB. Service fetches the file and then reads contents of the file and then manipulates the database based on file contents. What is the best way for spring rest service to fetch large file and operate upon its contents.

Upvotes: 0

Views: 3242

Answers (1)

Jay
Jay

Reputation: 9479

Use Apache File utils,

http://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/FileUtils.html#copyURLToFile(java.net.URL, java.io.File, int, int)

An example code to download 1 GB file from online is as below,

            package com.jai.download;

            import java.io.File;
            import java.net.URL;

            import org.apache.commons.io.FileUtils;

            public class FileDownload {

                public static void main(String[] args) {

                    File file = new File("c:/jj/1gb.zip");

                    URL url;
                    try {
                        url = new URL("http://download.thinkbroadband.com/1GB.zip");
                        long start = System.currentTimeMillis();
                        System.out.println("Downloading....");
                        FileUtils.copyURLToFile(url, file);
                        long end = System.currentTimeMillis();
                        System.out.println("Completed....in ms : " + (end - start));
                    } catch (Exception e) {
                        e.printStackTrace();
                    }

                }

            }

With my internet speed of 100 Mbps, it downloaded this 1GB file in 94 secs. Once you have your csv file like this downloaded you could read its contents to do you business logic check and make appropriate action.

Upvotes: 3

Related Questions