Why am I losing exactly 5 bytes during decoding in my netty pipeline?

Question

I'm writing a netty pipeline for a protocol which is able to transfer files and structured messages. File transfers are initiated with a structured message (handshake) followed by a stream of bytes representing the file.

The exact message flow for an incoming file looks like this (server being my netty implementation, client another software):

            +---------+                                    +---------+
            | Client  |                                    | Server  |
            +---------+                                    +---------+
                 |                                              |
                 | Connect (1)                                  |
                 |--------------------------------------------->|
                 |                                              |
                 | Handshake to announce incoming file (2)      |
                 |--------------------------------------------->|
                 |                                              |
                 |                Acknowledge file transfer (3) |
                 |<---------------------------------------------|
                 |                                              |
                 | Send file (4)                                |
                 |--------------------------------------------->|

The protocol message looks like this:

            +---------+----------------+----------------+
            | Length  |      Type      | Actual Message |
            | 4 bytes |     1 byte     |   N bytes      |
            +---------+----------------+----------------+

In the case of the handshake message, the Actual Message only consists of a single Long value, the token.

Here's the ReplayingDecoder:

    public class ProtocolDecoder extends ReplayingDecoder<State> {

      private Integer msgType;
      private Long token;


      public enum State {
        LENGTH,
        MSG_TYPE,
        TOKEN
      }

      public ProtocolDecoder() {
        super(State.LENGTH);
      }

      @Override
      protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {

        switch (state()) {

          case LENGTH:
            // (2) read the message length from the handshake
            long messageLength = in.readUnsignedIntLE();
            checkpoint(State.MSG_TYPE);

          case MSG_TYPE:
            // (2) read the message type from the handshake
            msgType = in.readBoolean();
            checkpoint(State.TOKEN);

          case TOKEN:
            try {
              // (2) read the token from the handshake
              token = in.readUnsignedIntLE();
              // (3) write back the acknowledgement
              ctx.channel().writeAndFlush(new Acknowledgement(token));
              // (4) done reading the protocol message
              // now switch to the file protocol
              ctx.pipeline().addLast(new FileInboundHandler());
              // write everything that is still in the buffer to the
              // modified pipeline
              ByteBuf rest = in.readBytes(super.actualReadableBytes());
              out.add(rest);
              // remove the protocol handshake decoder and pass
              // the rest of this channel to the `FileInboundHandler`
              ctx.pipeline().remove(this);
            } finally {
              reset();
            }
            break;
          default:
            throw new Error("Shouldn't reach here.");
        }
      }

      private void reset() {
        token = null;
        msgType = null;
      }

The FileInboundHandler simply creates a file and writes all ByteBufs to it.

This is working in principle, with the only problem being that each file misses exactly 5 bytes at the beginning.

I have 2 questions:

1) If I put a LoggingHandler as the first handler in the pipeline, can I be sure that all traffic on the socket is logged, no matter if my decoders are buggy?

2) When calling ctx.channel().writeAndFlush(), does it flush only the "outbound" buffer, meaning I can be sure I'm not flushing any bytes I have not consumed from the inbound buffer yet?

Answer 1

Could it be that you've forgotten to put a break; at the end of the statements for each switch case? Looking at it now, from the first LENGTH read it's gonna fall through to the type and token reads, getting 5 bytes too many: the boolean (one byte) and the token (read as unsigned int, 4 bytes).