Protobuf serialized object to json

Question

I have some standard objects that I am using too serialize using protobuf and store in a DB. Is it possible to deserialize the object into generic markup such as json or xml and back to the object type protobuf serialized WITHOUT having the concrete implementation of the class?

[ProtoContract]
[Serializable]
internal class SomeDTO
{
    /// <summary>
    /// Gets or sets the Id.
    /// </summary>
    [ProtoMember(1)]
    public string MyKey { get; set; }

    [ProtoMember(2)]
    public string Name { get; set; }

    [ProtoMember(3)]
    public int SomeId { get; set; }

    [ProtoMember(4)]
    public DateTime LastModifiedDate { get; set; }
}

This object would serialize into:

{
    "1": "asdklfhkajd",
    "2": "Tim",
    "3": 12345,
    "4": "05/29/2015 5:50"
}

or if I descriptors come over but not required:

{
    "MyKey": "asdklfhkajd",
    "Name": "Tim",
    "SomeId": 12345,
    "LastModifiedDate": "05/29/2015 5:50"
}

I was looking at possibly writing something with ProtoReader possibly?

Answer 1

Metadata Object:

public class collectionMetadata
{
    public string Field { get; set; }
    public string FieldType { get; set; }
}

Deserializer:

    public static JObject ProtoReaderDeserilizer(List<CollectionFieldType> collectionMetadata, ProtoReader reader)
    {
        JObject obj = new JObject();

        while (reader.ReadFieldHeader() > 0)
        {
            var field = collectionMetadata[reader.FieldNumber - 1].Field;
            switch (reader.WireType)
            {
                case WireType.Variant:
                    obj[field] = reader.ReadInt64();
                    break;
                case WireType.String:
                    var fieldType = collectionMetadata[reader.FieldNumber - 1].FieldType;

                    switch (fieldType.ToLowerInvariant())
                    {
                        case "date":
                            var tok1 = ProtoReader.StartSubItem(reader);
                            reader.ReadFieldHeader();

                            switch (reader.WireType)
                            {
                                case WireType.Variant:
                                    obj[field] = reader.ReadInt64();
                                    break;
                                default:
                                    reader.ReadFieldHeader();
                                    break;
                            }

                            ProtoReader.EndSubItem(tok1, reader);
                            break;

                        case "datetime":
                            obj[field] = BclHelpers.ReadDateTime(reader);
                            break;

                        case "decimal":
                            obj[field] = BclHelpers.ReadDecimal(reader);
                            break;

                        case "guid":
                            obj[field] = BclHelpers.ReadGuid(reader);
                            break;

                        case "string":
                        default:
                            if (!fieldType.StartsWith("["))
                                obj[field] = reader.ReadString();
                            else
                            {
                                var tok2 = ProtoReader.StartSubItem(reader);
                                obj[field] = ProtoReaderDeserilizer(JsonConvert.DeserializeObject<List<CollectionFieldType>>(fieldType), reader);
                                ProtoReader.EndSubItem(tok2, reader);
                            }
                            break;
                    }

                    break;
                case WireType.Fixed32:
                    obj[field] = reader.ReadSingle();
                    break;
                case WireType.Fixed64:
                    obj[field] = reader.ReadDouble();
                    break;
                case WireType.StartGroup:
                    // one of 2 sub-object formats
                    var tok = ProtoReader.StartSubItem(reader);
                    obj[field] = ProtoReaderDeserilizer(JsonConvert.DeserializeObject<List<CollectionFieldType>>(collectionMetadata[reader.FieldNumber - 1].FieldType), reader);
                    ProtoReader.EndSubItem(tok, reader);
                    break;
                default:
                    reader.SkipField();
                    break;
            }
        }

        return obj;
    }

Serializer:

    public static void ProtoSerializer(ProtoWriter writer, JObject obj, List<CollectionFieldType> collectionMetadata, string name)
    {
        int i = 1;

        if (obj == null)
        {
            throw new FormatException($"Collection {name} has invalid object defined in its field types. Ensure your collection schema and field times schema match.");
        }

        foreach (var field in collectionMetadata)
        {
            string exType = string.Empty;

            if (obj.TryGetValue(field.Field, out var fieldToken))
                exType = SerializeField(writer, name, i, field, exType, fieldToken);

            if (!string.IsNullOrEmpty(exType))
                throw new FormatException($"Collection: {name}, Field: {field.Field} invalid {exType} value: {fieldToken.ToString()}");

            i++;
        }
    }

    private static void SerializeDefaultValue(ProtoWriter writer, int i, CollectionFieldType field)
    {
        switch (field.FieldType.ToLowerInvariant())
        {
            case "bool":
                ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                ProtoWriter.WriteBoolean(default, writer);
                break;
            case "byte":
                ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                ProtoWriter.WriteByte(default, writer);
                break;
            case "sbyte":
                ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                ProtoWriter.WriteSByte(default, writer);
                break;
            case "decimal":
                ProtoWriter.WriteFieldHeader(i, WireType.Fixed64, writer);
                BclHelpers.WriteDecimal(default, writer);
                break;
            case "double":
                ProtoWriter.WriteFieldHeader(i, WireType.Fixed64, writer);
                ProtoWriter.WriteDouble(default, writer);
                break;
            case "float":
                ProtoWriter.WriteFieldHeader(i, WireType.Fixed64, writer);
                ProtoWriter.WriteDouble(default, writer);
                break;
            case "int":
                ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                ProtoWriter.WriteInt32(default, writer);
                break;
            case "enum":
                ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                ProtoWriter.WriteInt32(default, writer);
                break;
            case "long":
                ProtoWriter.WriteFieldHeader(i, WireType.Fixed64, writer);
                ProtoWriter.WriteInt64(default, writer);
                break;
            case "short":
                ProtoWriter.WriteFieldHeader(i, WireType.Fixed32, writer);
                ProtoWriter.WriteInt16(default, writer);
                break;
            case "date":
                ProtoWriter.WriteFieldHeader(i, WireType.String, writer);
                var dateToken = ProtoWriter.StartSubItem(field, writer);

                ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                ProtoWriter.WriteInt32(default, writer);

                ProtoWriter.EndSubItem(dateToken, writer);
                break;
            case "datetime":
                ProtoWriter.WriteFieldHeader(i, WireType.String, writer);
                BclHelpers.WriteDateTime(default, writer);
                break;

            case "guid":
                ProtoWriter.WriteFieldHeader(i, WireType.String, writer);
                BclHelpers.WriteGuid(default, writer);
                break;
            case "char":
            case "string":
            default:
                ProtoWriter.WriteFieldHeader(i, WireType.String, writer);
                ProtoWriter.WriteString(string.Empty, writer);
                break;
        }
    }

    private static string SerializeField(ProtoWriter writer, string name, int i, CollectionFieldType field, string exType, JToken fieldToken)
    {
        switch (field.FieldType.ToLowerInvariant())
        {
            case "bool":
                if (bool.TryParse(fieldToken.ToString(), out bool boolVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                    ProtoWriter.WriteBoolean(boolVal, writer);
                }
                else
                {
                    exType = "bool";
                }
                break;
            case "byte":
                if (byte.TryParse(fieldToken.ToString(), out byte byteVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                    ProtoWriter.WriteByte(byteVal, writer);
                }
                else
                {
                    exType = "byte";
                }
                break;
            case "sbyte":
                if (sbyte.TryParse(fieldToken.ToString(), out sbyte sbyteVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                    ProtoWriter.WriteSByte(sbyteVal, writer);
                }
                else
                {
                    exType = "sbyte";
                }
                break;
            case "decimal":
                if (decimal.TryParse(fieldToken.ToString(), out decimal decimalVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.Fixed64, writer);
                    BclHelpers.WriteDecimal(decimalVal, writer);
                }
                else
                {
                    exType = "decimal";
                }
                break;
            case "double":
                if (double.TryParse(fieldToken.ToString(), out double doubleVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.Fixed64, writer);
                    ProtoWriter.WriteDouble(doubleVal, writer);
                }
                else
                {
                    exType = "double";
                }
                break;
            case "float":
                if (float.TryParse(fieldToken.ToString(), out float floatVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.Fixed64, writer);
                    ProtoWriter.WriteDouble(floatVal, writer);
                }
                else
                {
                    exType = "float";
                }
                break;
            case "enum":
            case "int":
                if (int.TryParse(fieldToken.ToString(), out int intVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                    ProtoWriter.WriteInt32(intVal, writer);
                }
                else
                {
                    exType = "int";
                }
                break;
            case "long":
                if (long.TryParse(fieldToken.ToString(), out long longVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.Fixed64, writer);
                    ProtoWriter.WriteInt64(longVal, writer);
                }
                else
                {
                    exType = "long";
                }
                break;
            case "short":
                if (short.TryParse(fieldToken.ToString(), out short shortVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.Fixed32, writer);
                    ProtoWriter.WriteInt16(shortVal, writer);
                }
                else
                {
                    exType = "short";
                }
                break;
            case "date":
                ProtoWriter.WriteFieldHeader(i, WireType.String, writer);
                var dateToken = ProtoWriter.StartSubItem(fieldToken, writer);

                if (int.TryParse(fieldToken.ToString(), out int dateVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.Variant, writer);
                    ProtoWriter.WriteInt32(dateVal, writer);
                }
                else
                {
                    exType = "date";
                }

                ProtoWriter.EndSubItem(dateToken, writer);
                break;
            case "datetime":
                if (fieldToken.Type == JTokenType.Date)
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.String, writer);
                    BclHelpers.WriteDateTime((DateTime)fieldToken, writer);
                }
                else
                {
                    exType = "DateTime";
                }
                break;

            case "guid":
                if (Guid.TryParse(fieldToken.ToString(), out Guid guidVal))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.String, writer);
                    BclHelpers.WriteGuid(guidVal, writer);
                }
                else
                {
                    exType = "guid";
                }
                break;
            case "char":
            case "string":
            default:
                if (field.FieldType.StartsWith("["))
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.String, writer);
                    var token = ProtoWriter.StartSubItem(fieldToken, writer);
                    ProtoSerializer(writer, fieldToken as JObject, JsonConvert.DeserializeObject<List<CollectionFieldType>>(field.FieldType), name);
                    ProtoWriter.EndSubItem(token, writer);
                }
                else
                {
                    ProtoWriter.WriteFieldHeader(i, WireType.String, writer);
                    ProtoWriter.WriteString(fieldToken.ToString(), writer);
                }
                break;
        }

        return exType;
    }

Answer 2

No, basically. Because protobuf (the binary format) is ambiguous, and without additional context on how to interpret fields (which it gets from the type, and other metadata), it is not possible to reliably deserialize data. Even things as simple as strings and integers can be readily confused with multiple types. Also, at the protobuf level, there is no concept of a date/time as a primitive.

To see what I mean: take a binary payload from what you have above and throw it at https://protogen.marcgravell.com/decode - it has to offer you multiple interpretations of most types, because of the multiple possibilities. And the set of options it gives you is not exhaustive.

You could deserialize with SomeDTO and then use any JSON serializer of your choosing to serialize it to JSON, but: you'll need the concrete type. I have considered adding options to do this using a .proto schema plus a reader, but that doesn't really change what you need - it just changes the shape of it.

---

If you know the layout but don't have the type, there are options, but at that point it is easier just to create the type.