CODEWITHSUNDEEP
groovyfile-writing
Eqbal
Eqbal

Reputation: 4880

Groovy: Appending to large files

How do I append to large files efficiently. I have a process that has to continually append to a file and as the file size grows the performance seems to slow down as well. Is there anyway to specify a large buffer size with the append

Upvotes: 6

Views: 17939

Answers (3)

eckes
eckes

Reputation: 10423

In the JVM on Windows the append flag has been implemented inefficently with a seek operation.

This is neighter atomic nor very performant when opening the file multiple times. It is supposed to be fixed somewhere in the Java VM 7: https://bugs.java.com/bugdatabase/view_bug?bug_id=6631352

Upvotes: 0

robbbert
robbbert

Reputation: 2193

While Don's approach is valid, in general (it will throw an exception, though, because of a syntax error, and you need to flush() a BufferedOutputStream), I've been planning to elaborate further (which takes its time).

Groovy does not provide special objects for I/O operations. Thus, you would use Java's FileOutputStream (for writing bytes) or FileWriter (for writing strings). Both provide a constructor that takes a boolean append parameter.
For both, there exist decorators (BufferedOutputStream and BufferedWriter), which are buffered. "Buffering", in this scope, means, that contents are not necessarily written instantly to the underlying stream, and thus, there's a potential for I/O optimization.

Don already provided a sample for the BufferedOutputStream, and here's one for the BufferedWriter:

File file = new File("foo")
if (file.exists()) {
    assert file.delete()
    assert file.createNewFile()
}

boolean append = true
FileWriter fileWriter = new FileWriter(file, append)
BufferedWriter buffWriter = new BufferedWriter(fileWriter)

100.times { buffWriter.write "foo" }

buffWriter.flush()
buffWriter.close()

While Groovy does not provide its own I/O objects, the Groovy JDK (GDK) enhances several Java types by adding convenience methods. In the scope of I/O outputting, the OutputStream and the File types are relevant.

So, finally, you can work with those the "Groovy way":

new File("foo").newOutputStream().withWriter("UTF-8") { writer ->
    100.times { writer.write "foo" + it }
}

EDIT: As per your further inquiry:

None of the GDK methods allows for setting a buffer size.

The above "Groovy" code will overwrite the file if called repeatedly. - In contrast, the following piece of code will append the string to an existing file and, thus, can be called repeatedly:

new File("foo").withWriterAppend("UTF-8") { it.write("bar") }

Upvotes: 14

Dónal
Dónal

Reputation: 187539

def file = new File('/path/to/file')

// Create an output stream that writes to the file in 'append' mode
def fileOutput = new FileOutputStream(file, true)

// Buffer the output - set bufferSize to whatever size buffer you want to use
def bufferSize = 512
def fileOutput = new BufferedOutputStream(fileOutput, bufferSize)

try {
    byte[] contentToAppend = // Get the content to write to the file
    fileOutput.write(contentToAppend)
} finally {
    fileOutput.close()
}

Upvotes: 0

Related Questions