Convert IMFMediaBuffer data having data type of YUY2 to RGB24 or RGB32

Question

I am reading a frames from my web cam using MediaFoundation APIS.

IMFMediaType mediatype = null;
Hresult hr= mSourceReaderAsync.GetNativeMediaType((int)MF_SOURCE_READER.FirstAudioStream, i, out mediatype); 

returns only YUY2 media types.So i am getting the output of ReadSample gives YUY2 frame. I need to convert YUY2 to RGB24 or BitmapSource to show in WPF window. This is my OnRead callback method

 public HResult OnReadSample(HResult hrStatus, int dwStreamIndex, MF_SOURCE_READER_FLAG dwStreamFlags, long llTimestamp, IMFSample pSample)
{
    HResult hr = hrStatus;
    IMFMediaBuffer pBuffer = null;
    Stream s = null;
    JpegBitmapDecoder jpgdecoder = null; 
    BitmapSource cameraframe = null; 
    lock (this)
    {
        try
        {
            if (Succeeded(hr))
            {
                if (pSample != null)
                {
                    // Get the video frame buffer from the sample.
                    hr = pSample.GetBufferByIndex(0, out pBuffer);
                }
            }
            if (pBuffer != null)
            {
                int maxlen, curlen;
                pBuffer.GetMaxLength(out maxlen);
                pBuffer.GetCurrentLength(out curlen);
                var arr = new byte[maxlen - 1];
                pBuffer.Lock(out IntPtr ptr, out int maxLen, out int curLen);
                if (arr == null)
                    arr = new byte[maxlen - 1];
                var writable = (maxlen > 0) ? true : false;
                if (s == null)
                    s = new MemoryStream(arr, writable);

                System.Runtime.InteropServices.Marshal.Copy(ptr, arr, 0, curlen);


                s.Flush();
                s.Seek(0, SeekOrigin.Begin);
                if (jpgdecoder == null)
                    jpgdecoder = new JpegBitmapDecoder(s, BitmapCreateOptions.None, BitmapCacheOption.OnLoad);

                var frame = jpgdecoder.Frames[0];
                cameraframe = frame;
            }
            dispatcher.Invoke(() =>
            {
                OnCapture.Invoke(this, cameraframe);
            });
            // Request the next frame.
            if (Succeeded(hr))
            {
                // Ask for the first sample.
                
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.ToString());
        }
        finally
        {
            SafeRelease(pBuffer);
            SafeRelease(pSample);
            dispatcher.Invoke(() =>
            {
                hr = mSourceReaderAsync.ReadSample((int)MF_SOURCE_READER.FirstVideoStream, 0, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero);
            });
        }
    }

    return hr;
}

now it raises exception that {"No imaging component suitable to complete this operation was found."}

Answer 1

You have several options for converting frames.

1. On your CPU. I do not recommend this for obvious reasons, but if the image preview is not your main feature, you can get away with this. Extract the buffer like you are doing and convert the data array manually, preferably with array operations.
2. On hardware. I have two options that come to mind. The first is OpenGL. Assign your pixel buffer to a buffer in OpenGL and with a shader, you can convert the pixels from yuy2 to RGB. To do so, use the yuy2 spec.
3. The other option on hardware is using MFTs. I do not really know why you do not want to do this; it is fast. I see that you are implementing this in C#, which is quite the challenge, but I have done this before, proving it is possible. I can add a small code sample if you prefer this, but I suggest figuring this out with the docs. The key is to configure your transforms, then start the transform (set the pipeline to take in frames), and then pipe buffers from the samples you read into the transform, after which you extract the result from the transform.

I would recommend MFTs in your scenario. Maybe try to set up DirectX or OpenGL for displaying; your solution with BitmapSources can be slow for higher resolutions.

Answer 2

I think you can use Media foundation transform to convert between types.You can refer samples from MF.net by snarfle enter link description here