CODEWITHSUNDEEP
javafile-ionetty
Trevor Freeman
Trevor Freeman

Reputation: 7232

How to handle a large stream that is mostly written to disk in netty (use ChunkedWriteHandler?)?

I have a binary protocol with some unknown amount of initial header data (unknown length until header is fully decoded) followed by a stream of data that should be written to disk for later processing.

I have an implementation that decodes the header and then writes the file data to disk as it comes in from one or more frames in ChannelBuffers (so my handler subclasses FrameDecoder, and builds the message step by step, not waiting for one ChannelBuffer to contain the entire message, and writing the file data to disk with each frame). My concern is whether this is enough, or if using something like ChunkedWriteHandler does more than this and is necessary to handle large uploads.

Is there a more optimal way of handling file data than just writing it directly to disk from the ChannelBuffer with each frame?

Upvotes: 0

Views: 1005

Answers (2)

Norman Maurer
Norman Maurer

Reputation: 211

You should also think about adding an ExecutionHandler in front if your Handler. This will help you to not get blocked by the disk I/O. Otherwise you may see slow downs on heavy disk access.

Upvotes: 1

trustin
trustin

Reputation: 12351

It should be enough as long as the throughput is good enough. Otherwise, you might want to buffer the received data so that you don't make system calls too often (say, 32KiB bounded buffer) when the amount of received data is not large enough.

Netty could be even faster if it exposed the transferTo/From operation to a user, but such a feature is not available yet.

Upvotes: 2

Related Questions