CODEWITHSUNDEEP
protocol-buffersgrpcproto
Sivalingam
Sivalingam

Reputation: 931

How to achieve dynamic custom fields of different data type using gRPC proto

Looking for a solution in gRPC protobuff to implement dynamic fields of different datatypes for an multi-tenant application.

Also there can be any number of dynamic fields based on tenant.

Using map in proto, I can define different set of map for each data type. Is there any optimized way to achieve this.

Any help on this is appreciated.

Upvotes: 6

Views: 11856

Answers (1)

Eric Anderson
Eric Anderson

Reputation: 26464

There are a few different ways of transferring dynamic content in protobuf. Which is ideal varies depending on your use case. The options are ordered by their dynamism. Less dynamic options normally have better performance.

Use google.protobuf.Any in proto3. This is useful when you want to store arbitrary protobuf messages and is commonly used to provide extension points. It replaces extensions from proto2. Any has a child message and its type, so your application can check at runtime if it understands the type. If your application does not know the type, then it can copy the Any but can't decode its contents. Any cannot directly hold scalar types (like int32), but each scalar has a wrapper message that can be used instead. Because each Any includes the type of the message as a string, it is poorly suited if you need lots of them with small contents.

Use the JSON mapping message google.protobuf.Value. This is useful when you want to store arbitrary schemaless JSON data. Because it does not need to store the full type of its contents, a Value holding a ListValue of number_values (doubles) will be more compact on-the-wire than repeated Any. But if a schema is available, an Any containing a message with repeated double will be more compact on-the-wire than Value.

Use a oneof that contains each permitted type. Commonly a new message type is needed to hold the oneof. This is useful when you can restrict the schema but values have a relationship, like if the position of each value in a list is important and the types in the list are mixed. This is similar to Value but lets you choose your own types. While technically more powerful than Value it is typically used to produce a more constrained data structure. It is equal to or more compact on-the-wire than Value. This requires knowing the needed types ahead-of-time. Example: map<string, MyValue>, where MyValue is:

message MyValue {
  oneof kind {
    int32 int_value = 1;
    string string_value = 2;
  }
}

Use a separate field/collection for each type. For each type you can have a separate field in a protobuf message. This is the approach you were considering. This is the most compact on-the-wire and most efficient in memory. You must know the types you are interested in storing ahead of time. Example: map<string, int32> int_values = 1; map<string, string> string_values = 2.

Upvotes: 20

Related Questions