SeNS
SeNS

Reputation: 184

C# green screen removal (chroma key technique) and combining background image/video with webcam stream

I'm working on an app where I need to overlay a webcam video of people on a green/blue background screen (shades of green/blue may differ) with a still background image or video.

Unfortunately, I can't find any professional solution (in the form of C# framework/library) to solve chroma key image filtering issue.

I decided to write my own solution using the OpenCV library. Using @Burak's solution for Python, I partially solved the issue by porting the code to C#:

private void Worker_DoWork(object sender, DoWorkEventArgs e)
{
    Mat mask, result = null;
    var worker = (BackgroundWorker)sender;
    while (!worker.CancellationPending)
    {
        using (var frame = videoCap.RetrieveMat())
        {
            // Get a chroma key mask
            mask = ChromaKeyMask(frame);

            if (videoBack != null)
            {
                if (videoBack.Width != frame.Width && videoBack.Height != frame.Height)
                {
                    videoBack = videoBack.Resize(frame.Size());
                }

                result = videoBack.Clone();

                frame.CvtColor(ColorConversionCodes.RGB2BGR).CopyTo(result, mask);
            }

            Dispatcher.Invoke(() =>
            {
                FrameImage.Source = frame.ToWriteableBitmap();
                MaskImage.Source = mask.ToWriteableBitmap();
                ResultImage.Source = result?.CvtColor(ColorConversionCodes.BGR2RGB).ToWriteableBitmap();

                mask?.Dispose();
                result?.Dispose();
            });
        }

        Thread.Sleep(20);
    }
}

private Mat ChromaKeyMask(Mat src)
{
    Mat mask;
    using (var hsvMat = src.CvtColor(ColorConversionCodes.BGR2HSV))
    {
        var min = new Scalar(minH, minS, minV);
        var max = new Scalar(maxH, maxS, maxV);

        // Threshold the HSV image to extract green color
        mask = hsvMat.InRange(min, max);
        Cv2.BitwiseNot(mask, mask);

        // Smooth edges of the mask
        var size = new OpenCvSharp.Size(3, 3);
        var kernel = Cv2.GetStructuringElement(MorphShapes.Rect, size);
        Cv2.MorphologyEx(mask, mask, MorphTypes.Open, kernel);
        Cv2.GaussianBlur(mask, mask, size, 0, 0);
    }
    return mask;
}

This code is working but still not good enough for production grade: mask is "unstable" and "floating", and final image, after combining with background, has a green "halo".

app screenshot

So, my questions are:

I'm really sorry (I'm new to OpenCV and image processing), but I'm asking for answers with specific working solutions, not general thoughts about chroma key (yes, I've already read the Wikipedia article and watched a lot of questions/answers here on StackOverflow) . There is no need to use C#; Python and C++ will work fine.

Here you can find a modified @Barak Python script with images for test.

I tried a different SDKs from https://www.visioforge.com and https://www.graphicsmill.com: pretty unstable and/or simple non-working.

I need a professional grade chroma key filtering solution, to process webcam stream in realtime.

Upvotes: 0

Views: 363

Answers (0)

Related Questions