CODEWITHSUNDEEP
image-processingopticalflowvideo-codecs
cowboysaif
cowboysaif

Reputation: 241

Why don't we use motion vector data from video in optical flow?

All the implementation i saw in optical flow in opencv uses video as an array of frames and then implement optical flow on each image. That involves slicing the image into NxN block and searching for velocity vector.

Although motion vector in video codec is misleading and it does not necessarily contain motion information, why don't we use it to check which block likely has motion and then run optical flow on those blocks ? Shouldn't that fasten the process ?

Upvotes: 2

Views: 2860

Answers (3)

Pervez Alam
Pervez Alam

Reputation: 1256

Motion vector calculated for video encoding is not 'True Motion Vector'. The calculation targets to find best matching block to achieve highest compression. It doesn't targets the best motion estimation. That's why it can't be readily used for motion estimation purpose.

However, the motion-vectors from decoder can be used in some way to make motion estimation faster and better process. e.g.

  1. As a seed to your algorithm of motion estimation.
  2. Try filter like median (depending on data) to remove outlier motion vectors and use it for better motion estimation.
  3. One of above step can be used as first step in a two-step motion estimation algo.

Upvotes: 3

Stefan Karlsson
Stefan Karlsson

Reputation: 1085

I distinctly remember having read academic work done on the use of "motion" vectors embedded in h.264 and similar codecs for optical flow/vision motion analysis. The easiest approach to do so, is for iterative estimations (such as Horn and Schunk). One can simply bootstrap the iterative algorithm with the vectors from the decoder.

This will not improve the estimation accuracy, at best I would guess it could speed up the rate of convergence, allowing for earlier stopping.

Not all h.264 encoding is of the same quality either. Especially from real-time systems with limited hardware. There are likely cases where the motion vectors will be outright awful, and actually be a detriment to the flow estimations instead of helping. I am thinking for example of a low-end IP camera that is designed for mostly static scenes, that is moving in varying lighting etc.

Upvotes: 1

Marcus Müller
Marcus Müller

Reputation: 36347

OpenCV is a universal image processing framework. It takes in frames, not compressed video, for its algorithms.

You can certainly write a video decoder that also hands out info about displacement from the codec to openCV – but that will be very codec-specific and thus isn't in scope of openCV itself.

Upvotes: 2

Related Questions