Is there a way to workaround the limit of 255 types in a flatbuffers union?

Question

I am using flatbuffers to serialize rows from sql tables. I have a Statement.fbs that defines a statement as Insert, Update, Delete, etc. The statement has a member "Row" that is a union of all sql table types. However, I have more than 255 tables and I get this error when compiling with flatc:

$ ~/flatbuffers/flatc --cpp -o gen Statement.fbs
error: /home/jkl/fbtest/allobjects.fbs:773: 18: error: enum value does not fit [0; 255]

I looked through the flatbuffers code and I see that an enum is automatically created for union types and that the underlying type of this enum is uint8_t.

I do not see any options for changing this behavior.

I am able to create an enum that handles all my tables by specifying the underlying type to be uint16 in my flatbuffer schema file.

The statement schema:

include "allobjects.fbs";

namespace Database;

enum StatementKind : byte { Unknown = 0, Insert, Update, Delete, Truncate }

table Statement {
  kind:StatementKind;
  truncate:[TableKind];
  row:Row;
}

root_type Statement;

The allobjects Row union is a bit large to include here.

union Row {
    TypeA,
    TypeB,
    TypeC,
    Etc,
    ...
}

I suppose this is a design decision for flatbuffers that union types should only use one byte. I can accept that, but I would really like a workaround.

Answer 1

The nested buffer solution to the 255 limit of unions is pretty straight forward.

allobjects.fbs:

namespace Database;

table Garbage {
  gid:ulong;
  type:string;
  weight:uint;
}
... many more ...

Statement.fbs:

include "allobjects.fbs";

namespace Database;

enum StatementKind : byte { Unknown = 0, Insert, Update, Delete, Truncate }

// suppose this enum holds the > 255 Row types
enum TableKind : uint16 { Unknown = 0, Garbage, Etc... }

// this is the "union", but with a type enum beyond ubyte size
table Row {
  kind:TableKind;
  // this payload will be the nested flatbuffer
  payload:[ubyte];
}

table Statement {
  kind:StatementKind;
  truncate:[TableKind];
  row:Row;
}

root_type Statement;

main.c:

#include <iostream>
#include "Statement_generated.h"

void encodeInsertGarbage(unsigned long gid,
                         const std::string& type,
                         unsigned int weight,
                         std::vector<uint8_t>& retbuf)
{
    flatbuffers::FlatBufferBuilder fbb;

    // create Garbage flatbuffer
    // I used the "Direct" version so I didn't have to create a flatbuffer string object
    auto garbage = Database::CreateGarbageDirect(fbb, gid, type.c_str(), weight);
    fbb.Finish(garbage);

    // make [ubyte] from encoded "Garbage" object
    auto payload = fbb.CreateVector(fbb.GetBufferPointer(), fbb.GetSize());

    // make the generic Row homebrewed union
    auto obj = Database::CreateRow(fbb, Database::TableKind_Garbage, payload);
    fbb.Finish(obj);

    // create the Statement - 0 for "truncate" since that is not used for Insert
    auto statement = Database::CreateStatement(fbb, Database::StatementKind_Insert, 0, obj);
    fbb.Finish(statement);

    // copy the resulting flatbuffer to output vector
    // just for this test program, typically you write to a file or socket.
    retbuf.assign(fbb.GetBufferPointer(), fbb.GetBufferPointer() + fbb.GetSize());
}

void decodeInsertGarbage(std::vector<uint8_t>& retbuf)
{
    auto statement = Database::GetStatement(retbuf.data());
    auto tableType = statement->row()->kind();
    auto payload = statement->row()->payload();
    // just using a simple "if" statement here, but a full solution
    // could use an array of getters, indexed by TableKind, then
    // wrap it up nice with a template function to cast the return type
    // like rowGet<Garbage>(payload);
    if (tableType == Database::TableKind_Garbage)
    {
        auto garbage = Database::GetGarbage(payload->Data());
        std::cout << "  gid: " << garbage->gid() << std::endl;
        std::cout << "  type: " << garbage->type()->c_str() << std::endl;
        std::cout << "  weight: " << garbage->weight() << std::endl;
    }
}

int main()
{
    std::vector<uint8_t> iobuf;
    encodeInsertGarbage(0, "solo cups", 12, iobuf);
    decodeInsertGarbage(iobuf);
    return 0;
}

Output:

$ ./fbtest 
  gid: 0
  type: solo cups
  weight: 12

Answer 2

This sadly is a bit of a design mistake, and there is no workaround yet. Fixing this to be configurable is possible, but would be a fair bit of work given the amount of language ports that rely on it being a byte. See e.g. here: https://github.com/google/flatbuffers/issues/4209

Yes, multiple unions is a clumsy workaround.

An alternative could be to define the type as an enum. Now you have the problem that you don't have a typesafe way to store the table, though. That could be achieved with a "nested flatbuffer", i.e. storing the union value as a vector of bytes, which you can then cheaply call GetRoot on with the correct type, once you inspected the enum.

Another option may be an enum + a union, if the number of unique kinds of records is < 256. For example, you may have multiple row types that even though they have different names, their contents is just a string, so they can be merged for the union type.

Another hack could be to have declare a table RowBaseClass {} or whatever, which would be the type of the field, but you would never actually instantiate this table. You then cast back and forth to that type to store the actual table, dependending on the language you're using.