CODEWITHSUNDEEP
c#javaprotocol-buffersprotobuf-net
13Homer
13Homer

Reputation: 23

protobuf-net and repeated field

Is it right to use List as counterpart of the repeated field in protobuf? I'm trying this and always got exception:

Invalid wire-type; this usually means you have over-written a file without truncating or setting the length; see Using Protobuf-net, I suddenly got an exception about an unknown wire-type

The whole buffer is (the first msg, next which arrive are appended to this): 9 8 5 26 5 24 238 98 32 1

java protobuf file:

package XXX;

option java_package = "XXX";
option java_outer_classname = "Protos";

option optimize_for = SPEED;

message V3DDelta {
  optional int32 bid = 1;
  optional int32 bidSize = 2;
  optional int32 ask = 3;
  optional int32 askSize = 4;
}

message Request {
  optional int32 type = 1;
  optional string request = 2;
}

message Response {
  optional int32 type = 1;
  optional string response = 2;
  repeated V3DDelta v3dDelta = 3;
}

and protbuf-net classes:

[ProtoContract]
public class V3DDelta {
    [ProtoMember(1)]
    public double bid { get; set; }
    [ProtoMember(2)]
    public int bidSize { get; set; }
    [ProtoMember(3)]
    public double ask { get; set; }
    [ProtoMember(4)]
    public int askSize { get; set; }
}

[ProtoContract]
public class Request {
    [ProtoMember(1)]
    public int Type { get; set; }
    [ProtoMember(2)]
    public string Rq { get; set; }
}

[ProtoContract]
public class Response {
    [ProtoMember(1)]
    public int Type { get; set; }
    [ProtoMember(2)]
    public string Rsp { get; set; }
    [ProtoMember(3)]
    public List<V3DDelta> v3dDelta { get; set; }
    public Response() {
        v3dDelta = new List<V3DDelta>();
    }
}

I tried V3DDelta[] but result is the same. Reading message:

Response rsp = Serializer.DeserializeWithLengthPrefix<Response>(rcvstream, PrefixStyle.Base128);

and in java message is send using writeDelimitedTo. Buffer in c# is exactly the same as in java. When there was na v3dDelta field everything works as expected.

Upvotes: 2

Views: 10177

Answers (1)

Marc Gravell
Marc Gravell

Reputation: 1062502

Yes, either List<T> or an array (T[]) will work for repeated. Incidentally, there is a tool for generating protobuf-net classes from a .proto definition.

You are trying to read it "with length prefix", however: 9 is not valid as a varint prefix (9, as a field header, means "field 1, fixed 64-bit data", however: in this context it should be a varint).

Actually, none of your data is compatible with 9 as a field header, since you don't have any 64-bit values in your definition as field 1. You do have a double as field 3, which would do the job nicely - however, that would be 73 as a field header.

I would tell you what the sequence 9,8,5,... represents, however - we would have:

9 : field 1, fixed 64-bit
    8 5 26 5 24 238 98 32 <== payload for above
1 : field 0, fixed 64-bit
    ^^ not *really* valid, but fields <= 0 generally mean "stop" - but
       frankly this is not a clean/defined/expected exit condition

So again: please check your data. That doesn't look like a protobuf stream, or at least not one that matches your schema.

Edit: it occurred that maybe the java writeDelimitedTo includes just the length, without a header) making it technically not a consistent protobuf file, but... meh), so let's investigate that:

int len = ProtoReader.DirectReadVarintInt32(ms);

which gives us 9, and we have 9 bytes left, so looking good... 

8 : Field 1, varint
  5 = payload of above
26 : Field 3, length-delimited
  5 = length of payload
   24 238 98 32 1 = payload of ^^^
      24 : Field 3, varint
          238, 98 = payload of ^^^ = 12654
      32 : Field 4, varint
          1 = payload of ^^^ = 1

This now looks like it should parse... investigating why it isn't...

Edit 2: after a bit more debugging, part of this is because you (sorry to say) borked the V3DDelta properties. You defined them as int32 in the proto (and field 3 is a varint in the data), but you implemented them as double... and double and int32 are not friends.

So:

[ProtoContract]
public class V3DDelta
{
    [ProtoMember(1)]
    public int bid { get; set; }
    [ProtoMember(2)]
    public int bidSize { get; set; }
    [ProtoMember(3)]
    public int ask { get; set; }
    [ProtoMember(4)]
    public int askSize { get; set; }
}

and then the following works fine:

using (var ms = new MemoryStream(buffer))
{
    int len = ProtoReader.DirectReadVarintInt32(ms);
    var resp = (Response)model.Deserialize(ms, null, typeof(Response), len);

    Assert.AreEqual(5, resp.Type);
    Assert.AreEqual(1, resp.v3dDelta.Count);
    Assert.AreEqual(12654, resp.v3dDelta[0].ask);
    Assert.AreEqual(1, resp.v3dDelta[0].askSize);
}

Technically, I could make protobuf-net accept a varint for a double value, but this is highly indicative of a schema that doesn't really match, so I think the correct thing is to change the type in this case.

Upvotes: 2

Related Questions