Grpc.Core.RpcException "Failed to deserialize response message.." or "InvalidOperationException: Length mismatch"

Question

I am running a gRPC service on .NET Core 3.1 and trying to make calls from a .NET Framework 4.7.2 client. I'm using protobuf-net to reuse existing WCF data contracts. I've noticed the following unexpected client-side behavior today when one of the fields of a response object is non-null.

Grpc.Core.RpcException: 'Status(StatusCode="Internal", Detail="Failed to deserialize response message.")

• Packages: protobuf-net v2.4.4, Grpc v2.30.0, protobuf-net.Grpc v1.0.90, protobuf-net.Grpc.Native v1.0.90, Google.Protobuf v3.12.2

Here is an example that illustrates the general structure of the data contracts - in this case, Response<PersonData> is the response and PersonDataList is the non-null field.

   [DataContract]  
   public class Response<TValue>{
         [DataMember(Order = 1)]
         public TValue Value; 
    }

   [DataContract]  
   public class PersonData : Data {
         [DataMember(Order = 1)]
         public IList<PersonDataItem> PersonDataList; 
    }
    
    [DataContract]
    public PersonDataItem {
         [DataMember(Order = 1)] 
         public PersonDataType Type {get; private set;}
    
         [DataMember(Order = 2)] 
         public DateTime? Time {get; private set;}
    
         ....
         [DataContract]
         public enum PersonDataType : int {
    
               [EnumMember]
               Child = 1, 
               [EnumMember]
               Adult = 2
         }
    }
    
    [DataContract]
    [ProtoInclude(1, typeof(PersonData)]
    public class Data {
         [DataMember(Order = 1)]
         public string Name
    }

What stumps me is that I use a similar or same pattern in other data contracts, which throw no exceptions when deserializing the response. I did some searching and found this issue from 2019 that points to different Google.Protobuf versions as a possible source of error (but that doesn't seem to be the case here).

Has anyone seen this exception before? I'm not sure if this is an issue with my data contracts or perhaps with some package version mismatch. Any ideas or suggestions are very appreciated!

I also attempted updating to protobuf-net v3.0.0 but got a new client-side exception for every client call: Grpc.Core.RpcException: 'Status(StatusCode="Unknown", Detail="Exception was thrown by handler. InvalidOperationException: Length mismatch; calculated '63', actual '58'"...) This seems to be a different issue, and my current guess is that it may be related to the breaking changes re: dynamic typing in protobuf-net v3.

Answer 1

If you are using app.UseHttpLogging() on the client. Then, when receiving a response from the server, the client will issue such an error. This is one of the reasons for this error.

solution to this error:

builder.UseWhen(
            ctx => ctx.Request.ContentType != "application/grpc",
            builder =>
            {
                builder.UseHttpLogging();
            }
        );

This was discussed on the official microsoft github page. Link to discussion

Answer 2

There are two causes for this error

1. If you are using IIS to host the gRPC server there are some features that are not yet supported in ISS hence using Kestrel instead should fix the bug.
2. Explained by @vivekDev there is a bug in Visual Studio 2022 preview and turning off Hot Reload For Css should fix the bug

Answer 3

I am using Visual Studio 2022 Preview and I am facing this problem.

As mentioned here, there is a bug in Visual Studio 2022.

I applied the following work around mentioned and it worked for me. See the screen shots below.

Set 'Hot Reload CSS changes' to false as follows.

Answer 4

Frankly, this sounds like a bug in protobuf-net that you should log on GitHub, ideally with a repro that also shows the service contract (interface).

The types should look reasonable. The length mismatch at the end is particularly alarming, but I do not believe that this has any relationship to the dynamic types discussion.

Happy to help investigate here (I'm the author), but: this sounds more like a GitHub thing.

I suspect that the length difference is related to Data not being a proto-contract (so it is questionable as to whether the proto-include is being respected; this is reinforced by the var that there would be duplicate fields 1 if it were), but a proper repro would really help.

The following seems to work - the only changes to your code is to make sure that Data is marked as a ProtoContract, use a different number for the "include" part, and to add a Create method to PersonDataItem for the test harness to use; it is hard to know how much of the problem this accounts for without a repro.

using ProtoBuf;
using ProtoBuf.Meta;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.Serialization;

[DataContract]
public class Response<TValue>
{
    [DataMember(Order = 1)]
    public TValue Value;
}

[DataContract]
public class PersonData : Data
{
    [DataMember(Order = 1)]
    public IList<PersonDataItem> PersonDataList;
}

[DataContract]
public class PersonDataItem
{
    public static PersonDataItem Create(PersonDataType type, DateTime? time)
        => new PersonDataItem { Type = type, Time = time };
    [DataMember(Order = 1)]
    public PersonDataType Type { get; private set; }

    [DataMember(Order = 2)]
    public DateTime? Time { get; private set; }


    [DataContract]
    public enum PersonDataType : int
    {

        [EnumMember]
        Child = 1,
        [EnumMember]
        Adult = 2
    }
}

[ProtoContract, DataContract]
[ProtoInclude(10, typeof(PersonData))]
public class Data
{
    [ProtoMember(1), DataMember(Order = 1)]
    public string Name;
}

static class P
{
    static void Main()
    {
        var resp = new Response<PersonData>
        {
            Value = new PersonData
            {
                Name = "abc",
                PersonDataList = new List<PersonDataItem>
                {
                    PersonDataItem.Create(PersonDataItem.PersonDataType.Adult, DateTime.Now),
                }
            }
        };
        var clone = RoundTrip(resp);
        Console.WriteLine(clone.Value.Name);
        var item = clone.Value.PersonDataList.Single();
        Console.WriteLine(item.Time);
        Console.WriteLine(item.Type);
    }
    static T RoundTrip<T>(T value)
    {
        var model = RuntimeTypeModel.Default;
        using var state = model.Measure<T>(value);
        using var ms = new MemoryStream();
        state.Serialize(ms); // we expect this to explode if there was a length mismatch
        ms.Position = 0;
        return model.Deserialize<T>(ms);
    }
}