CODEWITHSUNDEEP
javaprotocol-buffers
Jamie McCrindle
Jamie McCrindle

Reputation: 9214

Using protocol buffers for binary logging

We're thinking of using Protocol Buffers for binary logging because:

That said, it isn't obvious how we should go about it because the APIs tend to focus on creating whole objects, so wrapping a list of DataLogEntry as a repeated field in a DataLogFile would be what you'd do in messaging terms but what we really want is just to be able to write and then read a whole DataLogEntry out, appending it to the end of a file.

The first issue we've hit by doing that is that doing this (in a test:

        FileInputStream fileIn = new FileInputStream(logFile);
        CodedInputStream in = CodedInputStream.newInstance(fileIn);
        while(!in.isAtEnd()) {
            DataLogEntry entry = DataLogEntry.parseFrom(in);
            // ... do stuff
        }

Only results in 1 DataLogEntry being read from the stream. Without the isAtEnd, it never stops.

Thoughts?

Edit: I've switched to using entry.writeDelimitedTo and BidLogEntry.parseDelimitedFrom and that seems to work...

Upvotes: 7

Views: 4613

Answers (2)

bmargulies
bmargulies

Reputation: 100050

As of 2.4.0a, at least, this is easy. Write your message with writeDelimitedTo. No need to use the coded streams directly.

Upvotes: 4

Michael Barker
Michael Barker

Reputation: 14378

From my understanding of protocol buffers it does not support multiple messages in a single stream. So you will probably need to track the boundaries of the messages yourself. you can do this by storing the size of the message before each message in the log.

public class DataLog {

    public void write(final DataOutputStream out, final DataLogEntry entry) throws IOException {
        out.writeInt(entry.getSerializedSize());
        CodedOutputStream codedOut = CodedOutputStream.newInstance(out);
        entry.writeTo(codedOut);
        codedOut.flush();
    }

    public void read(final DataInputStream in) throws IOException {
        byte[] buffer = new byte[4096];
        while (true) {
            try {
                int size = in.readInt();
                CodedInputStream codedIn;
                if (size <= buffer.length) {
                    in.read(buffer, 0, size);
                    codedIn = CodedInputStream.newInstance(buffer, 0, size);
                } else {
                    byte[] tmp = new byte[size];
                    in.read(tmp);
                    codedIn = CodedInputStream.newInstance(tmp);
                }
                DataLogEntry.parseFrom(codedIn);
                // ... do stuff
            }
            catch (final EOFException e) {
                break;
            }
        }
    }
}

NB: I've used an EOFException to find the end of file, you may wish to use a delimiter or track the number of byte read manually.

Upvotes: 4

Related Questions