What is the best way to minimize the impact of lost packets on a realtime media stream sent over TCP?

Question

We've implemented an audio-video collaboration application on top of Silverlight, and are trying to tune it. One of the issues we're experiencing is an increase in stream latency whenever a packet is dropped: we have to wait for the packet loss to be detected, requested, and then for the lost packet to be resent. Of course, this plays hell with the consistency of our audio stream. (We'd switch over to UDP if we could, but Silverlight doesn't support that in-browser. We've also disabled the Nagle algorithm, so in general, as soon as we submit a byte[] array to be transmitted, it's transmitted, and in a single packet. I'm aware that TCP packet size != amount of data submitted, but with the Nagle algorithm disabled, it's close. And we have an adaptive jitter buffer, so we can deal with lost packets, but a lost packet over TCP/IP massively increases the amount of audio we need to buffer, and hence latency.)

So we're trying to optimize how we send our packets, to see if there's any way to reduce the impact of dropped packets. We currently have several competing solutions that we're thinking about implementing:

(1) We could try to make our packets larger. Currently, we send a mix of large (~1024 byte video) packets and small (~70 byte audio) packets over the same TCP stream. But we could multiplex the audio and video data together, i.e., by attaching some of our video data to our audio packets whenever there's room. This would make the individual packets somewhat larger, but would cut down on the total number of packets.

(2) We could split the audio and video into two separate TCP streams. This means that if the video stream stalled because of a lost packet, the audio stream wouldn't stall, and vice versa. Of course, it would slightly increase the overhead, and wouldn't cut down on the overall number of packets sent.

(3) We could inverse multiplex the audio into multiple separate TCP streams, and then reassemble them on the far side. This would effectively allow us to "fake" a single UDP style of packet delivery. If we had 8 audio streams, and one of them stalled because of a lost packet, the other streams would still be able to deliver their data on time, and all we'd have to do is deal with 1/8 of the audio packets being unavailable until the stalled stream caught back up. That's not ideal, of course, but it might result in a better experience than to have the entire stream stall, and not being able to play any packets until the lost packet is retransmitted.

Any thoughts on any of these possibilities? Any other suggestions? Or do we just need to code up all three, and then test them?

Answer 1

I second @Kevin Nisbet on the buffer (unfortunately). If you're using TCP instead of UDP, the buffer needs to be as large as it takes for the server get notified about the missing bytes and for them to reach the client.

Since TCP delivers data to the application as an ordered stream, when a packet gets lost, the stack cannot deliver any additional byte to the app until the ack reporting the missing bytes is sent to the server, processed and the bytes arrive on the client. Meanwhile, the only thing keeping your app running is the buffer. Do you know how long does it take for the round-trip, including processing?

Without Selective Ack, anything received after that lost byte is useless and needs to be retransmitted. The client will ack the last byte received and the server needs to re-retransmit everything from that point on.

With Selective Ack at least the server only needs to send the missing chunk, but the stack needs to wait for the chunk to arrive nonetheless. It can't give the data it has received so far to the app and then fill in the missing parts. That's what UDP does :) Maybe you should write to MS...

Coming from the network side, I cringe (a bit) about sending multiple copies of the same content. Is bandwidth plentiful in your application? Maybe some sort of redundancy (FEC or similar) is better than duplicating your content. Besides, if packet loss could be happening I don't think it would be wise to shove more traffic on the network. Is your computer running half-duplex? :-D

Answer 2

If you re-enabled the Nagle algorithm you would (i) let TCP send out maximally-sized buffers according to the path MTU rather than your own determination; (ii) accomplish your suggestion (1) of piggybacking audio and video packets; and (iii) reduce the total number of packets. The steady-state performance of a saturated TCP connection with and without the Nagle algorithm is the same so you don't lose anything except during initial window filling.

You should also run the largest socket send buffer you can possibly afford: at least 128k, or double or quadruple that if possible; and you should also use as large a socket receive buffer as possible, although a socket receive buffer > 64k has to be set before connecting the socket so the other end can be told about window scaling during the TCP handshake.

Answer 3

How have you determined that it is packet loss that is causing the stalls?

I don't think separating the streams will help much, you'll just have more problems trying to keep the audio / video in sync.

Either way, no matter what tweaks you use, you will be limited by TCP/IP requiring the packet to be retransmitted. I guess the biggest thing I would look into is whether the TCP stacks on you're server and clients have some of the more advanced options enabled. I'm specifically referring to Selective Acknowledgements and Fast retransmissions (any modern OS should have these by default). Fast retransmissions will have the client ask for a missing packet very quickly when it's detected missing, and the selective acknowledgements will have the server only retransmit the missing portions of the stream.

Ultimately though, it sounds as if you're not using a large enough jitter buffer, if you're unable to tolerate a single packet lost. It's also possible you're application isn't all that consistent in the timing used to send data to the tcp stack. I'd get some packet captures and try and get a good idea what's going on in the network, and see what you can do from there.

Answer 4

Is this application to be used over the Internet? Is the reason for the lost packets due to Internet quality? If so, beyond developing the application to be as fault tolerant as possible, you may also want to make sure the Internet circuits are of acceptable quality. Good Internet circuits today should not have any more than 0.1% packet loss. You can test Internet circuits and ISPs using our Packet Loss tool. It's free to use so help yourself.