FFmpeg memory growing in avcodec_decode_video2

Question

I am using C# to call the ffmpeg lib. I use the example code to decode an video in the MP4 format.

However, I found out that the memory grows a lot (like 10MB), when I called the avcodec_decode_video2 function.

The memory keeps growing until I called the avcodec_close(&CodecContext) function.

I suppose the reason may be that AVCodecContext alloc 10MB data during the call of avcodec_decode_video2 function. And this memory will not be released until the AVCodecContext is closed.

I do not know if I am right.

Main codes are in the following.

private bool readNextBuffer(int index)
    {
        /// Read frames into buffer
        //m_codecContext->refcounted_frames = 0;
        int frameNumber = 0;
        FrameBuffer[index].Clear();
        GC.Collect();

        while (frameNumber < m_bufferSize)
        {
            AVPacket Packet; /// vedio packet 
            AVPacket* pPacket = &Packet;
            //AVFrame* pDecodedFrame = FFmpegInvoke.avcodec_alloc_frame(); 
            AVFrame* pDecodedFrame = FFmpegInvoke.av_frame_alloc(); 
            if (FFmpegInvoke.av_read_frame(m_pFormatContext, pPacket) < 0)
            {
                Console.WriteLine("The end of the vedio.");
                break;
            }
            if (pPacket->stream_index == pStream->index)
            {
                /// Decode vedio frame
                int gotPicture = 0;
                int size = FFmpegInvoke.avcodec_decode_video2(m_codecContext, pDecodedFrame, &gotPicture, pPacket);
                if (size < 0)
                {
                    Console.WriteLine("End of the vedio.");
                    //throw new Exception(string.Format("Error while decoding frame {0}", frameNumber));
                    break;
                }

                if (gotPicture == 1)
                {
                    /// Allocate an AVFrame structure

                    if (convertFrame(pDecodedFrame, index))
                    {
                        frameNumber++;
                    }
                    else
                    {
                        Console.WriteLine("Error: convert failed.");
                    }
                }
            }
            FFmpegInvoke.av_frame_free(&pDecodedFrame);
            FFmpegInvoke.av_free_packet(pPacket);
            GC.Collect();
        }
        nowFrameIndex = 0;
        return FrameBuffer.Count > 0 ? true : false;
    }

Answer 1

Your break statements cause loss of the allocated frame, in both cases, and the packet, in only the second case. Other than that it looks good at first sight. 10MB isn't much depending on the type of video. E.g., 1080p video is 1920x1080x1.5=3MB/frame, and you'll typically have a few references being kept alive throughout the lifetime of the video, so at any time you'll have 5-10 frames alive=15-30MB for 1080p. If you have threading enabled (on by default), this grows by another 1 frame per extra active thread (a typical machine will have e.g. 4 threads active).

You didn't mention anything about memory usage growth over video lifetime. E.g., does it grow linearly over time while the video is playing (even 100s or 1000s of frames)? That would be an issue. You would normally use valgrind if this were c/c++, but maybe c# has other memory profiling tools.

As for your decoder loop, you're not reading caches frames after the demuxer signaled EOF, i.e. you're not feeding NULL packets into the decoder. This is recommended.