CODEWITHSUNDEEP
pythonflatbuffers
user1519665
user1519665

Reputation: 511

Fast Python Serialize to Flatbuffer?

While exploring the flatbuffer library for fast serialization, I noticed that the library had an incredibly fast way to read flatbuffer vectors into numpy arrays with the 'Variable'AsNumpy() command, but I have been unable to find (in source) a corresponding encoding method for serializing numpy to flatbuffer.

So far, I am seemingly stuck with their example:

for i in reversed(range(0, 10)):
  builder.PrependByte(i)

This is obviously not ideal. In reverse, one can simply call toNumpy() on most data vectors and that works great.

Is there something simple I'm missing or is this functionality just not available?

Upvotes: 4

Views: 3749

Answers (2)

Amit Sharma
Amit Sharma

Reputation: 2067

We can do it as follows. Consider we want to write bytesOfImage = testImage.tobytes() without using the PrependByte().

We can follow below steps:

  1. Make sure builder has been initialized correctly with StartVector.

    Image.ImageStartDataVector(builder, len(bytesOfImage))

    This will move head by len(bytesOfImage) of bytes or more depends on alignment operation etc. But we don't need to worry about this as StartVector API will take care of these things. We only need to know latest head after StartVector() call.

  2. Seek the header to correct place before writing into Bytes array.

    builder.head = builder.head - len(bytesOfImage)

    StartVector moves the head to new position, and as we know flatbuffers writes data in little endian order ie [N, N-1, N-2,.....0] way. So we need to go back to position len(bytesOfImage) from current updated head before write.

  3. Copy your data into Bytes Array

    builder.Bytes[builder.head : (builder.head + len(bytesOfImage))] = bytesOfImage

  4. Call EndVector() to make sure head moved to correct place for future writes.

    data = builder.EndVector(len(bytesOfImage))

Upvotes: 4

user1519665
user1519665

Reputation: 511

See this stackoverflow link for workaround and to monitor if the feature is updated:

https://github.com/google/flatbuffers/issues/4668#issuecomment-372430117

Upvotes: 1

Related Questions