CODEWITHSUNDEEP
performancetcpcompressionclient-server
BullyWiiPlaza
BullyWiiPlaza

Reputation: 19195

Most efficient way of sending (easily compressible) data over a TCP connection

I developed a TCP server in C/C++ which accepts connections by clients. One of the functionalities is reading arbitrary server memory specified by the client.

Note: Security is no concern here since client and server applications are ran locally only.

Uncompressed memory sending currently works as follows

  1. The client sends the starting address and end address to the server.
  2. The server replies with the memory read between the received starting and end address chunk-wise each time the sending buffer runs full.
  3. The client reads the expected amount of bytes (length = end address - starting address)

Sending large chunks of memory with a potentially high amount of 0 memory is slow so using some sort of compression would seem like a good idea. This makes the communication quite a bit more complicated.

Compressed memory sending currently works as follows

  1. The client sends the starting address and end address to the server.
  2. The server reads a chunk of memory and compresses it with zlib. If the compressed memory is smaller than the original memory, it keeps the compressed one. The server saves the memory size, whether it's compressed or not and the compressed bytes in the sending buffer. When the buffer is full, it is sent back to the client. The send buffer layout is as follows:

Total bytes remaining in the buffer (int) | memory chunks count (int) | list of chunk sizes (int each) | list of whether a chunk is compressed or not (bool each) | list of the data (variable sizes each)

  1. The client reads an int (the total bytes remaining). Then it reads the remaining buffer size using the remaining byte information. Now the client reads the memory chunks count (another int) to be able to parse the list of chunk sizes and the list of whether they are compressed or not. Then using the sizes as well as the compressed information, the client can access the list of data and apply a decompression if necessary. The raw memory buffer is then assembled from all the decompressed received data. Reading from the server continues till the expected amount of raw bytes is assembled.

My question is if the compression approach appears optimal or if I'm missing something important. Sending TCP messages is the bottleneck here so minimizing them while still transmitting the same data should be the key to optimize performance.

Upvotes: 0

Views: 793

Answers (1)

Marko Bencik
Marko Bencik

Reputation: 396

Hy, I will give you a few starting point. Remember those are only starting points.

First read this paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.156.2302&rep=rep1&type=pdf and this https://www.sandvine.com/hubfs/downloads/archive/whitepaper-tcp-optimization-opportunities-kpis-and-considerations.pdf

This will give you a hint what can go wrong and it is a lot. Basically my advice is concentrate on behavior of the server/network system. What I mean try to get it stress tested and try to get a consistent behavior.

If you get a congestion in the system have a strategy for that. Optimize the buffer sizes for the socket. Research how the ring buffers work for the network protocols. Research if you can use jumbo MTU. Test if jitter is a problem in your system. Often because of some higher power the protocols start behaving erratic ( OS is busy, or some memory allocation ).

Now the most important you need to stress test all the time every move you make. Have a consistent reproducable test with that you can test at any point.

If you are on linux setsockopt is your friend and enemy at the same time. Get to know how it works what it does.

Define boundaries what your server must be able to do and what not.

I wish you the best of luck. I'm optimizing my system for latency and it's tricky to say the least.

Upvotes: 1

Related Questions