CODEWITHSUNDEEP
javafile
hudi
hudi

Reputation: 16525

How to append/write huge data file text in Java

I have a database with 150k records. I want to write this to file as fast as possible. I've tried many approaches, but all seem slow. How do I make this faster?

I read these records in blocks of 40k. So first I read 40k then another 40k and so on.

After reading the records, this process returns a StringBuilder which contains 40k lines. Then we write this StringBuilder to a file. 

private static void write(StringBuilder sb, Boolean append) throws Exception {
    File file = File.createTempFile("foo", ".txt");

    FileWriter writer = new FileWriter(file.getAbsoluteFile(), append);
    PrintWriter out = new PrintWriter(writer);
    try {
        out.print(sb);           
        out.flush();
        writer.flush();
    } finally {
        writer.close();
        out.close();
    }
}

I read this other example but it is equally slow: Fastest way to write huge data in text file Java

I also tried it with NIO api:

private static void write(StringBuilder sb, Boolean append)) throws Exception {
    FileChannel rwChannel = new FileOutputStream("textfile.txt", true).getChannel();
    ByteBuffer bb = ByteBuffer.wrap(sb.toString().getBytes("UTF-8"));
    rwChannel.write(bb);
    rwChannel.close();
}

Which is the best method to write/append huge data into file?

Upvotes: 7

Views: 28725

Answers (5)

user207421
user207421

Reputation: 310883

You are opening the file, writing one line, then closing it. It's the opening and closing that takes the time here. Find a way to keep the output file open.

Upvotes: 0

Holger
Holger

Reputation: 298143

You don’t need a PrintWriter here. If you have whatever kind of Writer (e.g. a FileWriter) you can simply invoke append(sb) on it. And you don’t need to flush, close implies flushing.

private static void write(StringBuilder sb, Boolean append) throws Exception {
  File file = File.createTempFile("foo", ".txt");

  try(FileWriter writer = new FileWriter(file.getAbsoluteFile(), append)) {
      writer.append(sb);
  }
}

On my system I encountered a small performance improvement using a Channel rather than an OutputStream:

private static void write0a(StringBuilder sb, Boolean append) throws Exception {
  File file = File.createTempFile("foo", ".txt");

  try(Writer writer = Channels.newWriter(new FileOutputStream(
      file.getAbsoluteFile(), append).getChannel(), "UTF-8")) {
    writer.append(sb);
  }
}

However these are only slight improvements. I don’t see much possibilities here as all the code ends up calling the same routines. What could really improve your performance is keeping the Writer alive during the invocations and not flushing every record.

Upvotes: 11

Amr Gawish
Amr Gawish

Reputation: 2025

Did you try Apache IO, is the performance still the same?

Upvotes: -1

user1079877
user1079877

Reputation: 9368

If you have a huge amount of data, it's better that you don't store it to StringBuilder and then write it to file at once.

This is the best scenario:

1) Before you start process on the data create FileInputStream

FileOutputStream fos = new FileOutputStream("/path/of/your/file");

2) Create and OutputStreamWriter from this file

OutputStreamWriter w = new OutputStreamWriter(fos, "UTF-8");

3) Create BufferedWriter (Improve file writing performance)

BufferedWriter bw = new BufferedWriter(w);

4) Pass bw to your process function and then flush/close

bw.flush();
bw.close();

The functionality of StringBuilder and BufferedWriter is almost same, So you do not need to change your code so much. The only negative point of this scenario is that, your process will involve all the time that the data are writing to file, but if you don't process the data in different thread, it is not an issue.

In this way, it doesn't matter how large data is it

Upvotes: 6

Seelenvirtuose
Seelenvirtuose

Reputation: 20618

You are using a FileWriter (or a FileOutputStream in the second example). These are not buffered! So they write single chars resp. bytes to the disk.

That means, you should wrap the FileWriter in a BufferedWriter (or the FileOutputSystem in a BufferedOutputSystem).

private static void write(StringBuilder sb, Boolean append) throws Exception {
    File file = File.createTempFile("foo", ".txt");
    Writer writer = new BufferedWriter(new FileWriter(file.getAbsoluteFile(), append));
    PrintWriter out = new PrintWriter(writer);
    try {
        out.print(sb);           
        out.flush();
        writer.flush();
    } finally {
        writer.close();
        out.close();
    }
}

Upvotes: 1

Related Questions