CODEWITHSUNDEEP
c++flatbuffers
benjist
benjist

Reputation: 2881

FlatBuffers: How to write giant files

I'm experimenting with FlatBuffers to store OpenStreetMap data. So, this would be 50GB.

Since usually everything is kept in memory, how is it possible (if at all) to sequentially write data into a file?

I have a feeling this is not quite where FlatBuffers is good for.

Upvotes: 8

Views: 2978

Answers (2)

Vinicius Cesar
Vinicius Cesar

Reputation: 1

I think you could use file mapped memory. So the SO will manage the reads and writes (probably you will have to implement some allocators)

Upvotes: -3

Aardappel
Aardappel

Reputation: 6074

There currently is no way to create a single FlatBuffer without having it all in memory at once. The only way to do it is to instead write out a chain of (length-prefixed) smaller independent FlatBuffers.

Upvotes: 5

Related Questions