CODEWITHSUNDEEP
javasql
Rey Libutan
Rey Libutan

Reputation: 5314

Several FileOutputStreams at a time?

The situation is that:

  1. I have a csv file with records (usually 10k but up to 1m records)
  2. I will process each record (very basic arithmetic with 5 basic select queries to the DB for every record)
  3. Each record (now processed) will then be written to a file BUT not the same file every time. A record CAN be written to another file instead.

Basically I have 1 input file but several possible output files (around 1-100 possible output files).

The process itself is basic so I am focusing on how I should handle the records.

Which option is appropriate for this situation?

  1. Store several List s that will represent per possible output file, and then write each List one by one in the end?
  2. To avoid several very large Lists, every after processing each record, I will immediately write it to its respective output file. But this will require that I have streams open at a time.

Please enlighten me on this. Thanks.

Upvotes: 1

Views: 66

Answers (1)

Peter Walser
Peter Walser

Reputation: 15696

The second option is ok: create the file output streams on demand, and keep them open as long as it takes (track them in a Map for example).

The operating system may have a restriction on how many open file handles it allows, but those numbers are usually well beyond a couple hundreds of files.

A third option: You could also just append to files, FileOutputStream allows that option in the constructor:

new FileOutputStream(File file, boolean append)

This is less performant than keeping the FileOutputStreams open, but works as well.

Upvotes: 2

Related Questions