CODEWITHSUNDEEP
c#streamcompressiontpl-dataflow
TigeR
TigeR

Reputation: 35

Usage of DeflateStream and/or CryptoStream in TPL Dataflow TransformBlock

I'm using TPL Dataflow blocks to implement a packet based network protocol. This protocol is fixed and cannot be changed by me. There will mostly be small, but many packets.

I have a client component that connects to the server and reads raw packets. These raw packets are then posted as MemoryStreams to a BufferBlock and later, in a TransformBlock, decoded into packet structs depending on the packet type/id.

To send packets, this process is reversed with another chain of dataflow blocks. All of this works well as far as I can tell.

The problem is that these packets may or may not be compressed and may or may not be encrypted, based on server responses. I would solve this with new TransformBlocks put inbetween (using decompressing as an example):

static TransformBlock<Stream, Stream> CreateDecompressorBlock(ProtocolContext context)
{
    return new TransformBlock<Stream, Stream>(stream =>
    {
        if (!context.CompressionEnabled) return stream;
        return new DeflateStream(stream, CompressionMode.Decompress);
    }
}

This however seems to me like it is not the right way. As I understand it, the DeflateStream (and CryptoStream) decode the data as it is read. This means that the data is decoded when I'm decoding the packet contents, and not inside the TransformBlock itself, where only the wrapper Stream is created. This seems like it would circumvent the strengths of the Dataflow Blocks.

So another solution came to my mind where instead of returning a DeflateStream/CryptoStream, I read through it into another MemoryStream:

static TransformBlock<Stream, Stream> CreateDecompressorBlock(ProtocolContext context)
{
    return new TransformBlock<Stream, Stream>(stream =>
    {
        if (!context.CompressionEnabled) return stream;
        using (var deflate = new DeflateStream(stream, CompressionMode.Decompress))
        {
            var ms = new MemoryStream();
            deflate.CopyTo(ms);
            return ms;
        }
    }
}

This now seems like a waste of memory.

So my question is, is it sufficient to just wrap the stream and let the TransformBlock that decodes the packet contents later do the work, or should I use a bit more memory and then have maybe better separation and maybe parallelism? (Though I don't think that decoding will be the bottleneck, this will be the network).

Or is there a pattern I can use with TPL Dataflow that solves my problem even better?

Upvotes: 3

Views: 302

Answers (1)

i3arnon
i3arnon

Reputation: 116586

As always, this is a tradeoff and the decision can only be made by you.

I would go with the simplest solution which is just to connect both blocks, and the let the writing block "absorb" the added compression complexity since TDF blocks can increase their parallelism when they need to. But, I would only do that if the that block doesn't have a limited parallelism (MaxDegreeOfParallelism).

If there are limitations, then I would handle the actual compression in the compression block just as you described. This behavior can be done in a very high degree of parallelism and these memory buffer don't seem to be such a big issue over all.

If they are, you can add a buffer pool and so your buffer count would be limited to the block's MaxDegreeOfParallelism and you would only need to allocate these buffers once in the lifetime of your application.

Upvotes: 1

Related Questions