Real time video stream lag

Question

My application uses real time video stream that shows some presentation to users in real time(!). I have an ip camera and video encoder installed my computer and a server thats serves the video(ustream).

There is delay time that is near 30 seconds between serving video and real video.

This problem is insignificant problem for my situation. But in my situation all users must see same screen in same time. (no lag) Like a real time video game!

First question is why there is a lag(nearly 4 seconds!) between users ? Second question how can i get this lag time to zero or low-lag ?

Edit:

Stream Provider is ustream.com, h264 encode 360p 750fps

Answer 1

As it was mentioned before, your latency between what happens and what user see is collected from:

1. sensor latency. Perhaps microseconds
2. encoder latency. Usually we can speak about less than 100 ms
3. encoder software latency. If this piece is not broken, it is less than 100 ms
4. encoder-to-server network latency. Here it may deviate from 30 ms to 1000 ms. If there is any balancer, add more 500 ms
5. inner server latency. Here it may be from 10 ms to 500 ms. For example if server doesn't trust encoder and unpacks and reorder frames, it will be up to 200-500 ms
6. streaming protocol latency. If it is rtmp, it may be virtually 0 ms, with HLS it will be up to 30 000 ms.
7. server-client network latency. UDP gives smaller latency (with many many issues to discuss here of course), TCP gives bigger latency.
8. client software latency
9. client decoder latency
10. client video subsystem latency. If you do not know yet, but when you press key on a modern PC, it takes less time to go cross ocean and back than to draw a new letter on your monitor.

So I hope that you see how many things you have to optimize to get latency less than 1000 ms.

We have implemented WebRTC server and have achieved around 300 ms latency from IP camera to server and from server to browser. So regular plain browser gets video after 300 ms. Video is transferred from Moscow to Amsterdam and back.

WebRTC is really a good way to go for you with such requirements.

Answer 2

Let me answer both your questions:

"why is there a lag(nearly 4 seconds!) between users ?"

This is mostly due there's no time synchronization between the users in the player and that's not an easy task.

First of all let's consider what happens when a user starts a video at some time UTC:

1. The user obtains the stream information from the server (this could be a RTMP connection or a m3u8/dash manifest)
2. The user then starts to receive chunks of video, with every chunk starting from a keyframe - that is a full picture to which other frames refer to. A chunk can be started playing only from a keyframe.
3. When the video chunk is completed another one is played from the respective keyframe.

This is more or less what happens, no matter what kind of streaming it is.

So now when some other users, some seconds later, starts the same stream - they will open the same chunk (since that's the "newest") and start with the same keyframe, but this time it's already some seconds old.

The 4 seconds you're observing mean that ustream chunks are probably that exact length.

"how can i get this lag time to zero or low-lag ?"

I must state that I never tried synchronising video between multiple users but I did argue once with a colleague that it's possible and here's the approach I'd take:

1. First you need to synchronise the time between users (get to know the deviation between their computer clock and some absolute time, here's a good question on this
2. The player will tell you what is the current video time, out of that you need to calculate the deviation to server time.
3. The time deviation between playback and server time is the time that you should pause your video for.

Then if all goes well, all your users will see the video at exactly the same time (or at least below the 40 ms of a single frame duration).