How to encode/decode video using C#?

Question

A little background, I was given the task of fixing a few "small" bugs and maintaining this solution for streaming video across the network between two instances of our application. The solution was written by someone who is no longer here so there is some mystery in the code as well as some really fun pitfalls. The solution was written using ffmpeg with C++ code written to wrap the encoding/decoding related code as well as some of the streaming code. This C++ was then wrapped with SWIG so that it could interop with C# and pass the video frames up where they are rendered using VideoRendererElement which lives in a WPF control. The main reason the frames are passed up is because we have some custom protocols we need to send video data over and those are written using C# so as the video frames are passed up we wrap them in our own packets and send them out on the wire. This solution works and we can stream video using our custom protocols though it is something of a nightmare to maintain and work with.

My question is there a better way to go about this? I'm looking for ways to work at a lower level with the video data (in C#) so that I can take the video frames and package them in our own packets and send them out and be able to receive and rebuild the video on the other side. ffmpeg seems to be the common solution but I've run into a lot of issues with it and the GPL/LGPL thing I think is a problem.

The basic flow I'm looking to achieve, video file -> encode -> wrap in packet -> send over wire on protocol X -> get video data from packet -> decode -> render / save to disk

Answer 1

DirectShow is your friend. DirectShow is the low level layer used by most of the windows "multimedia" applications like Media Player, Audio Encoders, and so on.

Even if this library has been made for native developers you can access it from the managed world thanks to DirectShow.net. http://directshownet.sourceforge.net This is a well known and stable managed wrapper for DirectShow.

The only thing you have to do is to learn a little bit DirectShow to understand the concept of graphes and filters and then to create your own filters and graphes to use the power of DirectShow !

Answer 2

We are converting video files into various output formats (divx encoded avi, flv, mp4 etc.) for our mediadatabase application. Since we always worked with CLI-applications to do media conversion (talk about rasterizing EPS files to JPG using ImageMagick/GS), we relied heavily on FFMPEG-CLI.

In our special environment, we used "dumb" UNIX-servers as conversion machines (there is only sshd, ffmpeg, misc. ffmpeg libraries and samba installed). They are controlled through PuTTy's CLI from C# (WCF webservice) via SSH commands to do the real conversion.

The call for ffmpeg happens via ssh and is specialized for each TransformationType. The putty CLI is started through C#'s System.Diagnostics.Process namespace, events for output- and error-messages are handled for logging purposes.

The internet provides alot of resources concerning questions like "How can i convert mpg to flv using ffmpeg?", a little bit of research will help you out. Since we are talking about a copyright'd application, i cannot post complete code excerpts. But it should give you an architectural idea about a reliable, fast video encoding backend using C#.

Answer 3

In our project we using Microsoft Expression Encoder. It is not free. It can convert videos to different formats and sizes, extract thumbnails, etc.

Here is example:

using Microsoft.Expression.Encoder;

//...
//skiped
//...

MediaItem mediaItem = new MediaItem(videoToEncode.SourceFilePath);
mediaItem.ApplyPreset(PresetFilePath);

Job job = new Job();
job.ApplyPreset(PresetFilePath); // path to preset file, where settings of bit-rate, codec etc
job.MediaItems.Add(mediaItem);

job.EncodeProgress += OnProgress;
job.EncodeCompleted += EncodeCompleted;

job.DefaultMediaOutputFileName = "{OriginalFilename}.encoded.{DefaultExtension}";
job.CreateSubfolder = false;

job.OutputDirectory = videoToEncode.EncodedFilePath;
job.Encode();

Answer 4

You could also have a look at the various Microsoft Windows Media SDKs that are available for download. In a project a few years back we used the Windows Media Format SDK to extract thumbnails from uploaded video. These SDKs also have .NET sample code.

Answer 5

I wrote the VideoRendererElement back when there were no efficient ways to render video in WPF (v3.0). It uses some hackery to make it work.

If you want to simplify things a bit, drop the VRE and use the InteropBitmap for rendering (WriteableBitmap is ok, but not as efficient). Also drop SWIG and make your C++ dll a CLI/C++ dll, this way you can directly talk to your C++ from C# (and vice versa).

Another route you can go is to just create a DirectShow source filter that contains your transport/decoding stuff and you can use something like my WPF MediaKit to have it render into WPF (it uses the D3DImage. 0 hacks).

Also, don't be afraid of LGPL. As long as you keep it in it's own DLL and do not change the source, you are within the license restrictions.

Answer 6

I had all kinds of trouble using ffmpeg wrapped into a DLL. My video project was pretty simple - I just needed the converter to take a single thumbnail from a WMV.

After trying just what you describe, my solution was to just copy the ffmpeg.exe binary into my project as an external library. this also neatly gets around any code licensing issues, AFAIK...

        Guid temp = Guid.NewGuid();

        // just throw our ffmpeg commands at cmd.exe
        System.Diagnostics.ProcessStartInfo psi = 
            new System.Diagnostics.ProcessStartInfo("cmd.exe");

        psi.WorkingDirectory = Page.MapPath(@"~\Lib\ffmpeg.rev12665");

        psi.UseShellExecute = false;
        psi.RedirectStandardError = true;
        psi.RedirectStandardOutput = true;
        psi.RedirectStandardInput = true;

        System.Diagnostics.Process ps = System.Diagnostics.Process.Start(psi);

        StreamReader outputReader = ps.StandardOutput;
        StreamReader errorReader = ps.StandardError;
        StreamWriter inputWrite = ps.StandardInput;

        // uses extra cheap logging facility
        inputWrite.WriteLine("echo \"Ripping " + copiedFile + " " + 
            temp.ToString() + "\" >> log.txt");

        inputWrite.WriteLine("ffmpeg.exe -i \"" + copiedFile + 
            "\" -f image2 -vframes 1 -y -ss 2 tmp\\" + temp.ToString() + 
            ".jpg");

        inputWrite.WriteLine("exit");

        ps.WaitForExit(3000);

        if (ps.HasExited)
        {
            string thumbFile = Page.MapPath(@"~\Lib\ffmpeg.rev12665\tmp") + 
                @"\" + temp.ToString() + ".jpg";
            // ...
        }

Your ffmpeg command line might vary drastically from my example, but this is the most stable way I've found to get thumbnails out. Other stuff I found online regarding ffmpeg specifically did not have this solution (cmd.exe-based), but this is the only one I've gotten working well. Good luck!

Answer 7

What you might try looking at is SharpFFmpeg. It's licensed using the GPL, although you might be able to see how they wrote their wrapper, and you might be able to write your own, or get ideas about how to fix your current solution.

Edit:

There's a similar wrapper called ffmpeg-sharp over at code.google.com, which uses an LGPL - you can use this in commercial applications. I suspect both of these wrappers do much the same, although SharpFFmpeg is older and probably more mature.