To find optical flow as ndarray using cv2.calcOpticalFlowPyrLK()

Question

I need to find the optical flow between every 2 adjacent frames of a video using Lucas Kanade's optical flow. I'm using Python and openCV for the project.

As per my understanding, Lucas Kanade is a sparse method to find optical flow. Is there a dense implementation of it? If so, how to use it in Python?

Using cv2.calcOpticalFlowFarneback(), which is a dense method, we get as output an ndarray ('flow' in the below example) which contains the optical flow.

cv2.calcOpticalFlowFarneback(prev, next, pyr_scale, levels, winsize, iterations, poly_n, poly_sigma, flags[, flow]) → flow

Is there a way to get similar output using cv2.calcOpticalFlowPyrLK()?

cv2.calcOpticalFlowPyrLK(prevImg, nextImg, prevPts[, nextPts[, status[, err[, winSize[, maxLevel[, criteria[, flags[, minEigThreshold]]]]]]]]) → nextPts, status, err

When using cv2.calcOpticalFlowPyrLK() (above), the output obtained contains nextPts which contains the next points to track but it does not directly give the optical flow of the 2 frames. If I subtract the prevPts from the nextPts, is the result the optical flow between the two frames? I found ' prev(y,x)~next(y+flow(y,x)[1],x+flow(y,x)[0]) ' in the section explaining calcOpticalFlowFarneback() in this link: https://docs.opencv.org/2.4/modules/video/doc/motion_analysis_and_object_tracking.html hence the question. (The syntax for both cv2.calcOpticalFlowPyrLK() and cv2.calcOpticalFlowFarneback are also from this link) Below is my implementation of the same.

import cv2
import numpy as np
import os
import subprocess as sp

yuv_filename = 'can_0.yuv'
flow=[]

width, height = 320, 240

file_size = os.path.getsize(yuv_filename)
n_frames = file_size // (width*height*3 // 2)
f = open(yuv_filename, 'rb')


old_yuv = np.frombuffer(f.read(width*height*3//2), dtype=np.uint8).reshape((height*3//2, width))


# Convert YUV420 to Grayscale
old_gray = cv2.cvtColor(old_yuv, cv2.COLOR_YUV2GRAY_I420)


# params for ShiTomasi corner detection
feature_params = dict( maxCorners = 100,
                       qualityLevel = 0.3,
                       minDistance = 7,
                       blockSize = 7 )

# Parameters for lucas kanade optical flow
lk_params = dict( winSize  = (15,15),
                  maxLevel = 2,
                  criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))


p0 = cv2.goodFeaturesToTrack(old_gray, mask = None, **feature_params)


for i in range(1,n_frames):
    # Read Y, U and V color channels and reshape to height*1.5 x width numpy array
    yuv = np.frombuffer(f.read(width*height*3//2), dtype=np.uint8).reshape((height*3//2, width))

    # Convert YUV420 to Grayscale
    gray = cv2.cvtColor(yuv, cv2.COLOR_YUV2GRAY_I420)

    # calculate optical flow
    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, gray, p0, None, **lk_params)


    flow.append(np.subtract(p1,p0))
    good_old = p0[st==1]
    good_new = p1[st==1]

    
    # Now update the previous frame and previous points
    old_gray = gray.copy()
    p0 = good_new.reshape(-1,1,2)


f.close()

Is flow a list of ndarrays containing the optical flow between adjacent frames of the input video?

This is the output I got for flow[7] (randomly selected) but I'm not sure if these are optical flow values.

[[[ 6.7138672e-03  4.5318604e-03]]

 [[-1.6220093e-02  1.9645691e-03]]

 [[-8.5296631e-03  1.8482208e-03]]

 [[-5.8441162e-03  1.5701294e-02]]

 [[ 7.5836182e-03  2.3475647e-02]]

 [[-1.4129639e-02  1.6357422e-02]]

 [[ 4.4555664e-03  4.1809082e-03]]

 [[ 5.6457520e-04 -9.5863342e-03]]

 [[ 2.8991699e-04 -3.0517578e-05]]

 [[-2.3452759e-02 -1.5502930e-02]]

 [[-6.8283081e-03  3.3264160e-03]]

 [[-8.4381104e-03  7.7590942e-03]]

 [[ 5.7144165e-03  1.1177063e-02]]

 [[-1.4160156e-02  2.1179199e-02]]

 [[-1.0498047e-02  8.0099106e-03]]

 [[-1.8310547e-04  2.8953552e-03]]

 [[ 4.4937134e-03 -2.0904541e-03]]

 [[-4.7698975e-02  3.7708282e-02]]

 [[ 6.3323975e-03  1.3298035e-02]]

 [[-3.3233643e-02 -1.7229080e-02]]

 [[ 7.5683594e-03  2.4566650e-03]]

 [[-3.0364990e-03  3.4656525e-03]]

 [[-1.0345459e-02 -7.4539185e-03]]

 [[ 1.3168335e-02  2.1423340e-02]]

 [[-6.3476562e-03 -1.0681152e-02]]

 [[ 1.5869141e-03  1.0375977e-03]]

 [[ 2.1820068e-03  6.7329407e-03]]

 [[-9.6130371e-03  2.9449463e-03]]

 [[-2.1362305e-03  8.5525513e-03]]

 [[-1.7547607e-04  2.1362305e-04]]

 [[ 2.9144287e-03  1.4343262e-03]]

 [[ 2.9602051e-03 -7.1868896e-03]]

 [[-1.2878418e-02  5.0182343e-03]]

 [[-3.1585693e-03 -5.0544739e-05]]

 [[ 1.0070801e-03  1.3740540e-02]]

 [[ 6.7138672e-04  1.7852783e-03]]

 [[-2.0568848e-02 -1.2943268e-02]]

 [[-2.1057129e-03  4.5013428e-03]]]```

Also is there a way to get optical flow using the Lucas Kanade method such that it has the same size or dimensions as the input frames? If so how? 

Answer 1

You can estimate a dense flow field with the pure pyramidal Lucas Kanade method by computing for each pixel the corresponding point pairs. To do that:

• ignore cv2.goodFeaturesToTrack

• initialize point list p0 that contains all pixel locations of the image

  grid_y, grid_x = np.mgrid[0:prevImg.shape[0]:1, 0:prevImg.shape[1]:1]
  p0 = np.stack((grid_x.flatten(),grid_y.flatten()),axis=1).astype(np.float32)
  

• run the spare flow computation

  p1, status, err = cv2.calcOpticalFlowPyrLK(prevImg, currImg, p0, None)
  

• assemble dense flow field by computing and reshape motion vectors (p1 - p0) flow will have the shape (h x w x 2) and prevImg (h x w x 1)

  flow = np.reshape(p1 - p0, (prevImg.shape[0], prevImg.shape[1], 2))
  

However, this is not the most efficient way to compute a dense optical flow field. In OpenCV you may take a look to other DenseOpticalFlow methods (e.g. DIS, Farneback, RLOFDense, Dual-TVL1)