CODEWITHSUNDEEP
flatbuffers
user259819
user259819

Reputation: 149

How to stream process a flatbuffer bigger than RAM?

We have following scenario:

Our approach was to store meta information about the objects in [ObjectInfo] and binary concat all the image objects and put it into a ubyte vector.

But we have to manually track the pointer to the buffer and where we are within the objects vector. We do not profit from flatbuffer generated code to access the objects.

Schema example for problem:

root_type Container;

table Container {
    // Information about the big objects, has to fit into RAM,
    metaInfo:[ObjectInfo]
    // Does not fit into RAM, has to be streamed with arbitrary size buffer
    bigObjects:[Objects];
}

table ObjectInfo {
    type:int;  
    name:string;
    offset:double; // Offset within Objects
    length:double; // length of object
}

table Objects {
    objects:[ubyte] // Contains the objects, contatinated 
}

We need to partially process flatbuffers. What could we do?

Upvotes: 3

Views: 1298

Answers (1)

Aardappel
Aardappel

Reputation: 6074

There is no elegant way to do this with FlatBuffers, it would require a lot of hacks. You can't verify a partial buffer. You can't follow references to other tables safely, etc.

If you want to stream, put each piece of data in its own FlatBuffer that individually fits in ram, then stream a sequence of such buffers. You can use the SizePrefixed functionality to make these buffers easy to stream. You can use the file_identifier to recognize different kinds of buffers.

Also, don't use double for offset/length data, use an appropriately sized integer type.

Upvotes: 4

Related Questions