CODEWITHSUNDEEP
tcpgobuffer
agnsaft
agnsaft

Reputation: 1881

Go: efficiently handling fragmented data received via TCP

I am writing a small tcp-server that is only suppose to read data, parse the receiving data (dynamic length frames) and handling these frames. Consider the following code:

func ClientHandler(conn net.Conn) {
    buf := make([]byte, 4096)
    n, err := conn.Read(buf)
    if err != nil {
        fmt.Println("Error reading:", err.Error())
        return
    }
    fmt.Println("received ", n, " bytes of data =", string(buf))
    handleData(buf)

This is pretty much the essence of my code as is. I read X bytes into a empty buffer and then handle the data.

The problem occurs when:

  1. a frame of data is bigger than the buffer it is trying to read into from the tcp connection. In this case I cannot handle the data until I receive the rest (but then the receving buffer is already occupied by the previous data).
  2. when the buffer contains one and a half frame of data and I need to handle the first frame, while keeping the incomplete frame for later... in this case I need to remove the handled part from the buffer, and move the incomplete part so that there is room for the remainder of the frame.

Both scenarios would probably require reallocation and copying of data, which may perhaps be a costly operation? Furthermore, I have no ideas on how to handle frames that are larger than the buffer except for expanding the buffer... but an ever-increasing buffer may perhaps lead to performance issues and denial of service. Last, but not least, I do not know the standard library of Golang good enough to know if there are any package built explicitly for handling these situations.

So my questions are:

  1. is there any best practice for handling these situations?
  2. is there any golang package that will do some or most of this for me?

Thank you.

Upvotes: 2

Views: 2767

Answers (1)

Not_a_Golfer
Not_a_Golfer

Reputation: 49195

byte slices should support pretty optimized resizing (i.e. reserving more bytes than needed, not copying if they can, growing exponentially, the copying code is not written in go but part of the runtime, etc).

so you can use append and see how it works.

Another, more idiomatic approach, is using bufio.Reader to wrap the connection, and it will handle all this automatically. I've used it in a tcp server I've written and it was extremely fast. You just do:

r := bufio.NewReader(conn)

and then you can either read until a delimiter or a given amount of bytes if you know it in advance.

Upvotes: 6

Related Questions