CODEWITHSUNDEEP
goflatbuffers
J Aamish
J Aamish

Reputation: 532

Dynamic FlatBuffers delimiter

I'm using flatbuffer to send binary data over unix socket. The flatbuffer that I send is of dynamic length. The problem I'm facing is, how to know how many bytes I have to read for one table.

Is there something like a delimiter that can be appended while sending, which I can use to determine the end of the flatbuffer.

When I tried with a smaller size

    buf := make([]byte, 512)
    nr, err := c.Read(buf)
    if err != nil {
        fmt.Println("exit echo")
        return
    }

And if the flatbuffer that is bigger than 512 bytes is read, then this results in failure.

When I read by growing my buffer, then I'm not able to find the end of the read

var n, nr int
var err error
buf := make([]byte, 0, 4096) // big buffer
tmp := make([]byte, 512)
for {
    n, err = c.Read(tmp)
    if err != nil {
        break
    }
    nr += n
    if nr >= 4096 {
        err = errOverrun
        break
    }
    buf = append(buf, tmp[:n]...)
}
if err != nil {
    fmt.Println("read error:", err)
    break
}

Upvotes: 1

Views: 507

Answers (1)

Aardappel
Aardappel

Reputation: 6074

FlatBuffers does not include a length field by design, since in most context the length is an implicit part of the storage or transfer of a buffer.

If you have no way to know the size of a buffer, or you are streaming buffers, the best is to simply pre-fix any buffer with a 32bit length field, so you can use that to read the rest of the data.

In the C++ API this is even built-in (see SizePrefixed functions), but this hasn't been ported to Go yet, so you'd have to do it manually.

Upvotes: 2

Related Questions