background extraction and update from video using matlab

Question

I have a video about traffic scene. Now, I want to calculate the percentage of vehicle area on the road area (or the percentage of foreground area on the background area). The first step for this is background extraction. I have read many document and scientific articles about it, one of them recommends to use the mean filter following this formula:

This is the link of that the article. The results are very good, it is exactly what I want.

I followed his formula and I tried to write my code. But It didn't work! Who can help me and give me some advice. This is my code:

clc;    % Clear the command window.
close all;  % Close all figures (except those of imtool.)

imtool close all;  % Close all imtool figures.
clear;  % Erase all existing variables.
workspace;  % Make sure the workspace panel is showing.
fontSize = 14;
%input video;
step = 10;
vob = VideoReader('NKKN.avi');
frame = vob.read(inf);
vidHeight = vob.Height;
vidWidth = vob.Width;
nFrames = vob.NumberOfFrames;
%%% First-iteration background frame
background_frame = double(frame*0);
redbackground_frame = background_frame(:,:,1);
greenbackground_frame = background_frame(:,:,2);
bluebackground_frame = background_frame(:,:,3);
%calculate background
i = 0;
for k = 1:10 %get background from 10 frame (J=10)
    thisframe = double(read(vob, k));
    %background_frame = background_frame + thisframe;
    redbackground_frame = redbackground_frame + thisframe(:,:,1);
    greenbackground_frame = greenbackground_frame + thisframe(:,:,2);
    bluebackground_frame = bluebackground_frame + thisframe(:,:,3);
    i=i+1;
    disp(i);
end

A = redbackground_frame/i;
B = greenbackground_frame/i;
C = bluebackground_frame/i;
background = cat(3,A,B,C);
imshow(background);

Answer 1

Extracting the background of this video https://www.youtube.com/watch?v=URJxS1giCA4&ab_channel=AliShahzil

clear all
close all
reader = VideoReader('C:\Users\Ali Sahzil\Desktop\Media.wmv'); // your video file location
vid = {};
while hasFrame(reader)
    vid{end+1} = im2single(readFrame(reader));
end

bg = mean( cat(4, vid{:}), 4);

x =bg;
imshow( bg );

Answer 2

This is actually based on Shai's and user3725204's answers. I didn't use read and NumberOfFrames which are not recommended. I also adopted user3725204's suggestion, since there's no need to read all frames.

function backGrnd = getBackGrnd(filename, nTest, method)
    tic
    if nargin < 2, nTest = 20; end
    if nargin < 3, method = 'median'; end
    v = VideoReader(filename);
    nChannel = size(readFrame(v), 3);
    tTest = linspace(0, v.Duration-1/v.FrameRate , nTest);
    %allocate room for buffer
    buff = NaN([v.Height, v.Width, nChannel, nTest]);
    for fi = 1:nTest
        v.CurrentTime =tTest(fi);
        % read current frame and update model
        buff(:, :, :, mod(fi, nTest) + 1) = readFrame(v);
    end
    switch lower(method)
        case 'median'
            backGrnd = nanmedian(buff, 4);
        case 'mean'
            backGrnd = nanmean(buff, 4);
    end
    toc
end

And the result is like this:

subplot(221); imshow(uint8(TrafficVisionLab.getBackGrnd('traffic.avi', 10, 'mean')));
subplot(222); imshow(uint8(TrafficVisionLab.getBackGrnd('traffic.avi', 10, 'median')));
subplot(223); imshow(uint8(TrafficVisionLab.getBackGrnd('traffic.avi', 50, 'mean')));
subplot(224); imshow(uint8(TrafficVisionLab.getBackGrnd('traffic.avi', 50, 'median')));

Answer 3

Here is a very simple solution you can build upon. First you will need a sample background image of the scene with no traffic. We will call this 'bg'.

Here is a simple approach in pseudo-code:

      load in background image 'bg'
      set threshold upper value
      set threshold lower value
      loop until done for each frame
      {
        subtract 'bg' image from your first frame
        if pixel value of foreground > than threshold upper value
        {
          set foreground pixel value to 'nan'
        }
        if pixel value of foreground < than threshold lower value
        {
          set foreground pixel value to 'nan'
        }
        if pixel value of foreground == 0
        {
          set foreground pixel value to 'nan'
        }
      }

This will bracket your foreground images to only show the parts of the scene you are interested in. Note: this process can be greatly enhanced by using a stereoscopic camera to give you depth perception. However, you should be able to build upon this code to remove unwanted parts of your image.

Answer 4

vob = VideoReader('NKKN.avi');
frame = vob.read(inf);
vidHeight = vob.Height;
vidWidth = vob.Width;
nFrames = vob.NumberOfFrames;
%allocate room for buffer of 20 frames
buff = NaN( [vidHeight, vidWidth, 3, 20] ); % allocate room for buffer
for fi = 1:20:nFrames
disp(fi);
% read current frame
thisframe = double(read(vob, fi)) / 255; % convert to [0..1] range 
% update background model
buff(:, :, :, mod( fi, 10 ) + 1 ) = thisframe;
background_L1 = nanmedian( buff, 4 );
background_L2 = nanmean( buff, 4 );



hImage = subplot(2, 2, 1);
image(thisframe);
caption = sprintf('thisframe');
title(caption, 'FontSize', fontSize);
drawnow; % Force it to refresh the window.

subplot(2,2,2);
imshow(background_L2);
title('background-L2');

subplot(2,2,3);
imshow(background_L1);
title('background-L1');

end

Answer 5

You can maintain a buffer of B frames for a dynamic estimation of backgound

buff = NaN( [vidHeight, vidWidth, 3, B] ); % allocate room for buffer

% process the video
for fi = 1:nFrames
    % read current frame
    thisframe = double(read(vob, k)) / 255; % convert to [0..1] range 

    % update background model
    buff(:, :, :, mod( fi, B ) + 1 ) = thisframe;
    background_L1 = nanmedian( buff, 4 ); % I think this is better than `mean` - try it!
    background_L2 = nanmean( buff, 4 );

    % do whatever processing you need with fi-th frame 
    % and the current background mode...
    % ...
end

Note that if fi < B (i.e., you processed less than B frames) the background model is not stable. I am using NaNs as default values for the buffer and these values are ignored when backgound model is estimated -- this is the reason why I use nanmedian and nanmean instead of simply median and mean.