CODEWITHSUNDEEP
javaioniobytebuffer
Haywood Jablomey
Haywood Jablomey

Reputation: 47

Using Java's ByteBuffer to read millions of messages

Here's my problem: one big gzipped file; millions of messages.

Each message consists of:

***************** *************** ****************** 
* 2-byte LENGTH * * 1-byte TYPE * * N-byte PAYLOAD * , where N = (LENGTH-1).
***************** *************** ******************

Depending on the TYPE, I need to read a few bytes from an offset in the PAYLOAD and choose to accept or reject the message.

I know how to do this sort of thing using java.io.DataInputStream, but this seems like a perfect application of java.nio.ByteBuffer (see here!). However, I need some help setting it up.

So, how do I use ByteBuffer to read messages from my gzipped file?

Update

I guess what I'd like to see is a skeletal implementation of code that could get me on the right track to using ByteBuffer effectively. Thanks!

Upvotes: 3

Views: 1758

Answers (2)

Paul Gregoire
Paul Gregoire

Reputation: 9793

Instead of writing everything for your protocol at a low level, why not use a library like Mina or Netty? They can provide you with fairly easy to implement a solution.

Mina http://mina.apache.org/

Netty http://www.jboss.org/netty

As an aside, we use Mina in Red5 and we lots of implementers processing millions and millions of messages.

Upvotes: -1

Nick H
Nick H

Reputation: 11535

Sorry to not answer your question, but I'm not sure there's any benefit to using ByteBuffer over DataInputStream here. You'll probably want to put your stream through GZIPInputStream to inflate the data.

GZIPInputStream gzipInput = new GZIPInputStream(yourInputStream);
DataInputStream dataInput = new DataInputStream(gzipInput);

With ByteBuffer you'd probably just be wrapping the bytes read from your input stream, with no advantage.

Upvotes: 2

Related Questions