CODEWITHSUNDEEP
javacsvjava-8java-stream
Rishi Saraf
Rishi Saraf

Reputation: 1812

reading remote csv file without downloading

I have requirement to read remote big csv file line by line (basically streaming). After each read I want to persist record in db. Currently I am achieving it through below code but I am not sure if it download complete file and keep it in jvm memory. I assume it is not. Can I write this code in better way using some java 8 stream features

URL url = new URL(baseurl);
HttpURLConnection urlConnection = url.openConnection();
if(connection.getResponseCode() == 200)
{
     BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
     String current;
     while((current = in.readLine()) != null)
     {
        persist(current);
     }
 }

Upvotes: 4

Views: 3046

Answers (1)

Flown
Flown

Reputation: 11740

First you should use a try-with-resources statement to automatically close your streams when reading is done.
Next BufferedReader has a method BufferedReader::lines which returns a Stream<String>.
Then your code should look like this:

URL url = new URL(baseurl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if (connection.getResponseCode() == 200) {
  try (InputStreamReader streamReader = new InputStreamReader(connection.getInputStream());
      BufferedReader br = new BufferedReader(streamReader);
      Stream<String> lines = br.lines()) {
    lines.forEach(s -> persist(s)); //should be a method reference
  }
}

Now it's up to you to decide if the code is better and your assumption is right that you don't keep the whole file in the JVM.

Upvotes: 3

Related Questions