CODEWITHSUNDEEP
c++protocol-buffers
knip
knip

Reputation: 71

modify raw protobuf stream

Let's say I have compiled an application (Receiver) with the following proto file:

syntax = "proto3";  
message Control {   
    bytes version = 1;  
    uint32 id = 2; 
    bytes color = 3;
}

and I have another application (Transmitter) which initially has the same proto file but after an update a new field is added like:

syntax = "proto3";  
message Control {   
    bytes name = 1;  
    uint32 id = 2; 
    bytes color = 3;
    uint32 color_id = 4;
}

I have seen that if the Receiver app tries to parse the proto, change some data and then serialize it back the added fields coming from the Transmitter app are removed.

I need a way to change the id field directly accessing to the raw bytes without having to parse/serialize the proto. Is it possible ?

This is needed because I have some "header" fields in the Control message that I know that will never be changed but others that can be added/changed in the same proto of trasmitter app due to app update.

I have seen: https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.io.coded_stream but I was not able to modify an existing bytestream and the ReadString is not able to understand the string length.

Thanks in advance

Upvotes: 2

Views: 1387

Answers (3)

Marko Bencik
Marko Bencik

Reputation: 396

Just deserialize the entire message and map it on the new message. It is the cleanest way. You do not have a lot of data and probably no real time requirements. Create a mapper and do not overthink the problem.

Upvotes: 0

jpa
jpa

Reputation: 12176

In proto2 syntax, protobuf C++ library used to preserve unknown fields so that when you re-encoded the message, they would remain. Unfortunately this feature (like many others) have been removed in the proto3 syntax.

One workaround could be to do it this way:

  1. Set only the new id value in the Receiver message and encode it.
  2. Append this data after the original binary data.

This relies on the protobuf feature that appended messages replace original values of fields in protobuf messages.

Hmm, actually reading the issue report linked above, it seems that you can turn on unknown field preservation in protobuf version 3.5 and newer.

Upvotes: 0

dan
dan

Reputation: 333

I don't think there is an official way to do it. You could do this by hand following the encoding guidelines by protobuf (https://developers.google.com/protocol-buffers/docs/encoding#structure).

Basically you should do this:

  • start decoding with the very first bit
  • decode until you reach the field number of the id
  • identify the bits representing the id and replace them with your new (encoded!) id

This is bad for several reasons. Most importantly, your code has to know details about the message structure and content (field number and data type of your id), and this is exactly what you want to avoid when using protocol buffers (you always need some info from the .proto files).

Upvotes: 1

Related Questions