TheYapperTheBeest
Reputation: 31
Producing streamable video from screen using gdigrab and ffmpeg.autogen libav wrapper
I'm working on creating a class that captures the screen, encodes it using H.264, and outputs it in a streamable format to a pipe. For this, I'm using the ffmpeg.autogen version 5.1.2.3 (It is a libav wrapper in .net), along with the libav binaries from ffmpeg-n5.1-latest-win64-gpl-shared-5.1. Although I'm not an expert, I believe the following options are necessary to facilitate a streamable output: frag_keyframe, empty_moov, default_base_moof, and faststart.
The issue isn't with the receiving end, which I can confirm is functioning correctly. Despite not encountering any errors, the class I've written doesn't seem to produce a streamable output when tested with libVLC. There must be something wrong with the class implementation itself.
VLC OUTPUT:
Creating an input for 'imem://'
using timeshift granularity of 50 MiB
using timeshift path: C:\Users\kwist\AppData\Local\Temp
creating demux: access='imem' demux='any' location='' file='(null)'
looking for access_demux module matching "imem": 15 candidates
`imem://' gives access `imem' demux `any' path `'
Invalid get/release function pointers
no access_demux modules matched
using access module "imem_access"
looking for stream_filter module matching "prefetch,cache_read": 24 candidates
using 16777216 bytes buffer, 16777216 bytes read
using stream_filter module "prefetch"
looking for stream_filter module matching "any": 24 candidates
creating access: imem://
looking for access module matching "imem": 27 candidates
Trying Lua scripts in C:\Users\kwist\AppData\Roaming\vlc\lua\playlist
Trying Lua scripts in C:\Program Files\VideoLAN\VLC\lua\playlist
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\anevia_streams.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\anevia_xml.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\appletrailers.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\bbc_co_uk.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\cue.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\dailymotion.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\jamendo.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\koreus.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\liveleak.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\newgrounds.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\rockbox_fm_presets.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\soundcloud.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\twitch.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\vimeo.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\vocaroo.luac
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\playlist\youtube.luac
looking for stream_directory module matching "any": 1 candidates
no stream_filter modules matched
no stream_directory modules matched
attachment of directory-extractor failed for imem://
using stream_filter module "record"
looking for demux module matching "any": 55 candidates
creating demux: access='imem' demux='any' location='' file='(null)'
looking for stream_filter module matching "record": 24 candidates
looking for xml reader module matching "any": 1 candidates
using xml reader module "xml"
subtitle demux discarded
TS module discarded (lost sync)
MOD validation failed (ext=)
trying url: imem://
CPU flags: 0x000fd3db
this does not look like an MPEG PS stream, continuing anyway
using demux module "ps"
looking for meta reader module matching "any": 2 candidates
Trying Lua scripts in C:\Users\kwist\AppData\Roaming\vlc\lua\meta\reader
Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\reader\filename.luac
`imem://' successfully opened
garbage at input from 509, trying to resync...
no meta reader modules matched
Trying Lua scripts in C:\Program Files\VideoLAN\VLC\lua\meta\reader
The code:
using System;
using System.Diagnostics;
using System.IO;
using System.IO.Pipelines;
using System.Threading;
using System.Threading.Tasks;
using FFmpeg.AutoGen;
namespace DeltaFramesStreaming
{
public unsafe class ScreenStreamer : IDisposable
{
private readonly AVCodec* productionCodec;
private readonly AVCodec* screenCaptureAVCodec;
private readonly AVCodecContext* productionAVCodecContext;
private readonly AVFormatContext* productionFormatContext;
private readonly AVCodecContext* screenCaptureAVCodecContext;
private readonly AVDictionary* productionAVCodecOptions;
private readonly AVInputFormat* screenCaptureInputFormat;
private readonly AVFormatContext* screenCaptureInputFormatContext;
private readonly int gDIGrabVideoStreamIndex;
private readonly System.Drawing.Size screenBounds;
private readonly int _produceAtleastAmount;
private MemoryStream unsafeToManagedBridgeBuffer;
private CancellationTokenSource cancellationTokenSource;
private Task recorderTask;
private PipeWriter _pipeWriter;
public ScreenStreamer(int fps, int bitrate, int screenIndex, PipeWriter pipeWriter, int produceAtleastAmount = 1000)
{
_pipeWriter = pipeWriter;
ffmpeg.avdevice_register_all();
ffmpeg.avformat_network_init();
recorderTask = Task.CompletedTask;
cancellationTokenSource = new CancellationTokenSource();
unsafeToManagedBridgeBuffer = new MemoryStream();
_produceAtleastAmount = produceAtleastAmount;
// Allocate and initialize production codec and context
productionCodec = ffmpeg.avcodec_find_encoder(AVCodecID.AV_CODEC_ID_H264);
if (productionCodec == null) throw new ApplicationException("Could not find encoder for codec ID H264.");
productionAVCodecContext = ffmpeg.avcodec_alloc_context3(productionCodec);
if (productionAVCodecContext == null) throw new ApplicationException("Could not allocate video codec context.");
// Set codec parameters
screenBounds = RetrieveScreenBounds(screenIndex);
productionAVCodecContext->width = screenBounds.Width;
productionAVCodecContext->height = screenBounds.Height;
productionAVCodecContext->time_base = new AVRational() { den = fps, num = 1 };
productionAVCodecContext->pix_fmt = AVPixelFormat.AV_PIX_FMT_YUV420P;
productionAVCodecContext->bit_rate = bitrate;
int result = ffmpeg.av_opt_set(productionAVCodecContext->priv_data, "preset", "veryfast", 0);
if (result != 0)
{
throw new ApplicationException($"Failed to set options with error code {result}.");
}
// Open codec
fixed (AVDictionary** pm = &productionAVCodecOptions)
{
result = ffmpeg.av_dict_set(pm, "movflags", "frag_keyframe+empty_moov+default_base_moof+faststart", 0);
if (result < 0)
{
throw new ApplicationException($"Failed to set dictionary with error code {result}.");
}
result = ffmpeg.avcodec_open2(productionAVCodecContext, productionCodec, pm);
if (result < 0)
{
throw new ApplicationException($"Failed to open codec with error code {result}.");
}
}
// Allocate and initialize screen capture codec and context
screenCaptureInputFormat = ffmpeg.av_find_input_format("gdigrab");
if (screenCaptureInputFormat == null) throw new ApplicationException("Could not find input format gdigrab.");
fixed (AVFormatContext** ps = &screenCaptureInputFormatContext)
{
result = ffmpeg.avformat_open_input(ps, "desktop", screenCaptureInputFormat, null);
if (result < 0)
{
throw new ApplicationException($"Failed to open input with error code {result}.");
}
result = ffmpeg.avformat_find_stream_info(screenCaptureInputFormatContext, null);
if (result < 0)
{
throw new ApplicationException($"Failed to find stream info with error code {result}.");
}
}
gDIGrabVideoStreamIndex = -1;
for (int i = 0; i < screenCaptureInputFormatContext->nb_streams; i++)
{
if (screenCaptureInputFormatContext->streams[i]->codecpar->codec_type == AVMediaType.AVMEDIA_TYPE_VIDEO)
{
gDIGrabVideoStreamIndex = i;
break;
}
}
if (gDIGrabVideoStreamIndex < 0)
{
throw new ApplicationException("Failed to find video stream in input.");
}
AVCodecParameters* codecParameters = screenCaptureInputFormatContext->streams[gDIGrabVideoStreamIndex]->codecpar;
screenCaptureAVCodec = ffmpeg.avcodec_find_decoder(codecParameters->codec_id);
if (screenCaptureAVCodec == null)
{
throw new ApplicationException("Could not find decoder for screen capture.");
}
screenCaptureAVCodecContext = ffmpeg.avcodec_alloc_context3(screenCaptureAVCodec);
if (screenCaptureAVCodecContext == null)
{
throw new ApplicationException("Could not allocate screen capture codec context.");
}
result = ffmpeg.avcodec_parameters_to_context(screenCaptureAVCodecContext, codecParameters);
if (result < 0)
{
throw new ApplicationException($"Failed to copy codec parameters to context with error code {result}.");
}
result = ffmpeg.avcodec_open2(screenCaptureAVCodecContext, screenCaptureAVCodec, null);
if (result < 0)
{
throw new ApplicationException($"Failed to open screen capture codec with error code {result}.");
}
}
public void Start()
{
recorderTask = Task.Run(async () =>
{
AVPacket* packet = ffmpeg.av_packet_alloc();
AVFrame* rawFrame = ffmpeg.av_frame_alloc();
AVFrame* compatibleFrame = null;
byte* dstBuffer = null;
try
{
while (!cancellationTokenSource.Token.IsCancellationRequested)
{
if (ffmpeg.av_read_frame(screenCaptureInputFormatContext, packet) >= 0)
{
if (packet->stream_index == gDIGrabVideoStreamIndex)
{
int response = ffmpeg.avcodec_send_packet(screenCaptureAVCodecContext, packet);
if (response < 0)
{
throw new ApplicationException($"Error while sending a packet to the decoder: {response}");
}
response = ffmpeg.avcodec_receive_frame(screenCaptureAVCodecContext, rawFrame);
if (response == ffmpeg.AVERROR(ffmpeg.EAGAIN) || response == ffmpeg.AVERROR_EOF)
{
continue;
}
else if (response < 0)
{
throw new ApplicationException($"Error while receiving a frame from the decoder: {response}");
}
compatibleFrame = ConvertToCompatiblePixelFormat(rawFrame, out dstBuffer);
response = ffmpeg.avcodec_send_frame(productionAVCodecContext, compatibleFrame);
if (response < 0)
{
throw new ApplicationException($"Error while sending a frame to the encoder: {response}");
}
while (response >= 0)
{
response = ffmpeg.avcodec_receive_packet(productionAVCodecContext, packet);
if (response == ffmpeg.AVERROR(ffmpeg.EAGAIN) || response == ffmpeg.AVERROR_EOF)
{
break;
}
else if (response < 0)
{
throw new ApplicationException($"Error while receiving a packet from the encoder: {response}");
}
using var packetStream = new UnmanagedMemoryStream(packet->data, packet->size);
packetStream.CopyToAsync(unsafeToManagedBridgeBuffer).GetAwaiter().GetResult();
byte[] managedBytes = unsafeToManagedBridgeBuffer.ToArray();
_pipeWriter.WriteAsync(managedBytes).AsTask().GetAwaiter().GetResult();
unsafeToManagedBridgeBuffer.SetLength(0);
}
}
}
ffmpeg.av_packet_unref(packet);
ffmpeg.av_frame_unref(rawFrame);
if (compatibleFrame != null)
{
ffmpeg.av_frame_unref(compatibleFrame);
ffmpeg.av_free(dstBuffer);
}
}
}
finally
{
ffmpeg.av_packet_free(&packet);
ffmpeg.av_frame_free(&rawFrame);
if (compatibleFrame != null)
{
ffmpeg.av_frame_free(&compatibleFrame);
}
}
});
}
public AVFrame* ConvertToCompatiblePixelFormat(AVFrame* srcFrame, out byte* dstBuffer)
{
AVFrame* dstFrame = ffmpeg.av_frame_alloc();
int buffer_size = ffmpeg.av_image_get_buffer_size(productionAVCodecContext->pix_fmt, productionAVCodecContext->width, productionAVCodecContext->height, 1);
byte_ptrArray4 dstData = new byte_ptrArray4();
int_array4 dstLinesize = new int_array4();
dstBuffer = (byte*)ffmpeg.av_malloc((ulong)buffer_size);
ffmpeg.av_image_fill_arrays(ref dstData, ref dstLinesize, dstBuffer, productionAVCodecContext->pix_fmt, productionAVCodecContext->width, productionAVCodecContext->height, 1);
dstFrame->format = (int)productionAVCodecContext->pix_fmt;
dstFrame->width = productionAVCodecContext->width;
dstFrame->height = productionAVCodecContext->height;
dstFrame->data.UpdateFrom(dstData);
dstFrame->linesize.UpdateFrom(dstLinesize);
SwsContext* swsContext = ffmpeg.sws_getContext(screenCaptureAVCodecContext->width,
screenCaptureAVCodecContext->height,
(AVPixelFormat)screenCaptureAVCodecContext->pix_fmt,
productionAVCodecContext->width,
productionAVCodecContext->height,
productionAVCodecContext->pix_fmt,
ffmpeg.SWS_FAST_BILINEAR,
null,
null,
null);
ffmpeg.sws_scale(swsContext, srcFrame->data, srcFrame->linesize, 0, screenCaptureAVCodecContext->height, dstFrame->data, dstFrame->linesize);
ffmpeg.sws_freeContext(swsContext);
return dstFrame;
}
public void Dispose()
{
cancellationTokenSource.Cancel();
recorderTask.Wait();
fixed (AVCodecContext** pProductionAVCodecContext = &productionAVCodecContext)
{
ffmpeg.avcodec_free_context(pProductionAVCodecContext);
}
ffmpeg.avformat_free_context(productionFormatContext);
fixed (AVCodecContext** pScreenCaptureAVCodecContext = &screenCaptureAVCodecContext)
{
ffmpeg.avcodec_free_context(pScreenCaptureAVCodecContext);
}
ffmpeg.avformat_free_context(screenCaptureInputFormatContext);
unsafeToManagedBridgeBuffer?.Dispose();
cancellationTokenSource.Dispose();
}
private System.Drawing.Size RetrieveScreenBounds(int screenIndex)
{
return new System.Drawing.Size(1920, 1080);
}
}
}
I also conducted research on testing methods for writing output to an MP4 file, utilizing mp4box.exe to retrieve the atom boxes. You can view an 8-second sample at the following link:
https://pastebin.com/FNsqmHXm
Finally I'm wondering if there's a clear reason why this class isn't generating output that can be streamed, and what the solution might be.
I experimented with different movflags settings, including testing the dash option, hoping it would clarify the streaming module for VLC, with no success.
Upvotes: 3
Views: 176