CODEWITHSUNDEEP
c#protocol-buffers
Brett Woodward
Brett Woodward

Reputation: 408

Reading Protobuf TCP Packets with existing C# classes

This problem seems simple and do-able enough, but I cannot for the life of me get it to work.

I have:

All I want to do is parse those packets using what I know from the assembly file. Simple? Probably, but no matter what I do, nothing actually is getting parsed.

Here's an example of one of the many possible classes:

    [DebuggerNonUserCode, CompilerGenerated, GeneratedCode("ProtoGen", "2.4.1.473")]
    public sealed class Thing: GeneratedMessageLite<Thing, Thing.Builder>
    {
        // Fields
        private static readonly string[] _thingFieldNames = new string[] { "list" };
        private static readonly uint[] _thingFieldTags = new uint[] { 10 };

        ...

        public static Builder CreateBuilder()
        {
            return new Builder();
        }

        ...

        public static thing ParseFrom(ByteString data)
        {
            return CreateBuilder().MergeFrom(data).BuildParsed();
        }

        ...

        public override void WriteTo(ICodedOutputStream output)
        {
            int serializedSize = this.SerializedSize;
            string[] strArray = _thingFieldNames;
            if (this.list_.Count > 0)
            {
                output.WriteMessageArray<thingData>(1, strArray[0], this.list_);
            }
        }

        ...

        [DebuggerNonUserCode, GeneratedCode("ProtoGen", "2.4.1.473"), CompilerGenerated]
        public static class Types
        {
            // Nested Types
            [CompilerGenerated, GeneratedCode("ProtoGen", "2.4.1.473")]
            public enum PacketID
            {
                ID = 19
            }
        }
    }

There are many others like that. I've tried doing something like this with each packet (using protobuf-csharp-port):

    Console.WriteLine(Thing.ParseFrom(packet.Buffer).ToString());

I'm expecting to see the actual text data. But I either get nothing, an error about invalid packet tags, or an error about it being a "0".

I've also tried using protobuf-net, but it just gives me random errors about incompatibility, unexpected types, etc.:

    Console.WriteLine(ProtoBuf.Serializer.Deserialize<Thing>(ms));

What on Earth am I doing wrong here? Is there a better way to, using all the known types in an assembly, simply decode the Protobuf message and see what's inside? Ideally without having to know beforehand what type of message it is?

Thank you so much if you can figure this out!

Upvotes: 2

Views: 1573

Answers (1)

afrischke
afrischke

Reputation: 3866

Guessing from the failed attempts outlined in the question, I believe that you have some misconceptions about the content of your pcap file. This line in particular

Console.WriteLine(Thing.ParseFrom(packet.Buffer).ToString());

makes me think that you are working under the wrong assumption that a single pcap packet contains the serialized bytes of one single object. Unfortunately, this is not the case.

As you might know, TCP/IP networks use a layered protocol stack, where each layer adds functionality and isolates upper layer protocols from the details of lower layer protocols (and vice versa). This is done by encapsulating the data sent from the upper layers down to the network and de-encapsulating the data as it travels up the stack on the receiving side. Now, your pcap file contains the raw data as seen by your network interface, i.e. the serialized payload plus all the data added by the application, transport, internet, and link layer.

Now, if you want to de-serialize the objects contained in your dump, you will need to write some code that removes all the headers of the link layer and internet protocols, (un-)does the work of the transport protocol and reassembles the stream of bytes that was sent over the network.*

Next, you will need to analyze the resulting byte dump and make some sophisticated guesses about the design of the application level protocol. Does it implement a handshake when it starts communicating? Does it send a checksum together with the actual payload? Was the data compressed before it was sent over the network? Does the application encrypt the data prior to sending it? If TCP was used as the transport protocol, how is message framing implemented etc. Of course, if you have access to the source code of the application that generated the data (or at least the application binaries), then you can just read the code (or reverse engineer the binaries) to figure this part out.

Once you are at this point you are in a position to interpret the raw data. All that is left is to write some code that extracts the relevant bytes, feeds it to the protocol-buffer deserializer and voilà, you have your objects back!

(* And there are other minor issues like fragmented IP packets, TCP segments that arrived out of order, and TCP retransmissions, of course.)

To sum it all up:

  • It is theoretically possible to write a tool that de-serializes objects that were serialized using protocol-buffers from a pcap dump, provided that the dump contains the full communication between both peers, i.e. packets were not truncated by the tool that generated the dump etc.
  • In practice however, there are multiple hurdles to overcome that are anything but trivial even for a seasoned practitioner of the art, as such a tool must:
    1. be able to deal with all the complex issues of the lower level protocols of TCP/IP to rebuild the data flow between the peers.
    2. be able to understand the application level protocol that was used to transmit the serialized objects.

Note that point 1 above alone results in the requirement to implement the functionality of a TCP/IP stack at least in part. The easiest way to accomplish this would probably consist in reusing code of an open source TCP/IP implementation such as the one found in the Linux or *BSD kernel. Many tools that do similar things, like reconstructing HTTP traffic from capture files, do exactly this. (See e.g. Justsniffer.)

Upvotes: 5

Related Questions