ffmpeg (libav, libavfilter, etx) - modify frame with image or text using C\C++ API

Question

After reading a huge bunch of docs and tutorials I still cant find a way to add some image or text to each frame of video. Something like logo on the frame corner, or text watermark.

Iam know how to do such things with ffmpeg from cli, but for this case, C\C++ code is required.

Looks like, ffmpeg's libav allow me to do some things with frame on decode stage, using AVFrame structure of current frame and add some modifications to it with libavfilter. But how exactly this can be done?

Answer 1

If you're still looking for an answer to this, or anyone stumbles across this question in hopes of an answer, here's what I'd do.

1. Open the image that you want to use as a watermark.

    int ret = -1;
    ret = avformat_open_input(&imgFmtCtx_, filename, NULL, NULL);
    ret = avformat_find_stream_info(imgFmtCtx_, NULL);
    for(int i = 0; i < imgFmtCtx_->nb_streams; i++)
    {
        if(imgFmtCtx_->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO)
        {
            const AVCodec *imgDecoder = avcodec_find_decoder(imgFmtCtx_->streams[i]->codecpar->codec_id);
            imgDecCtx_ = avcodec_alloc_context3(imgDecoder);
            ret = avcodec_parameters_to_context(imgDecCtx_, imgFmtCtx_->streams[i]->codecpar);
            imgDecCtx_->framerate = av_guess_frame_rate(imgFmtCtx_, imgFmtCtx_->streams[i], NULL);
            imgDecCtx_->time_base = av_inv_q(imgDecCtx_->framerate);
            ret = avcodec_open2(imgDecCtx_, imgDecoder, NULL);
            break;
        }
    }

2. Initialize the filter graph of 2 buffers, 1 overlay, and 1 buffersink.

    snprintf(args, sizeof(args), 
    "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
        videoDecCtx->width, videoDecCtx->height, 
        videoDecCtx->pix_fmt,
        videoDecCtx->time_base.num, videoDecCtx->time_base.den, 
        videoDecCtx->sample_aspect_ratio.num, videoDecCtx->sample_aspect_ratio.den 
    );
    ret = avfilter_graph_create_filter(&bufferSrc0Ctx_, avfilter_get_by_name("buffer"), "in0", args, NULL, filterGraph_ );
    
    enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_YUVA420P, AV_PIX_FMT_NONE };
    ret = avfilter_graph_create_filter(&bufferSinkCtx_, avfilter_get_by_name("buffersink"), "out", NULL, NULL, filterGraph_);
    ret = av_opt_set_int_list(bufferSinkCtx_, "pix_fmts", pix_fmts, AV_PIX_FMT_NONE, AV_OPT_SEARCH_CHILDREN);

    snprintf(args, sizeof(args), 
    if(!width_) width_ = imgDecCtx_->width;
    if(!height_) height_ = imgDecCtx_->width;
    "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
        width_,height_,
        imgDecCtx_->pix_fmt,
        // imgDecCtx_->time_base.num,imgDecCtx_->time_base.den,
        videoDecCtx->time_base.num, videoDecCtx->time_base.den, // The two overlays need to have exact time-stamps. There might be a smarter way of rescaling the time_bases of the videoFrame and imgFrame but this works too.
        imgDecCtx_->sample_aspect_ratio.num, imgDecCtx_->sample_aspect_ratio.den
    );
    ret = avfilter_graph_create_filter(&bufferSrc1Ctx_, avfilter_get_by_name("buffer"), "in1", args, NULL, filterGraph_);

    snprintf(args, sizeof(args), "x=100:y=100");
    ret = avfilter_graph_create_filter(&overlayCtx_, avfilter_get_by_name("overlay"), "overlay", args, NULL, filterGraph_);

    ret = avfilter_link(bufferSrc0Ctx_, 0, overlayCtx_,     0);
    ret = avfilter_link(bufferSrc1Ctx_, 0, overlayCtx_,     1);
    ret = avfilter_link(overlayCtx_,    0, bufferSinkCtx_,  0);
    
    ret = avfilter_graph_config(filterGraph_, NULL);

3. Start a while loop, read in video frames and images, sync the time-stamps of the video frame and image frame, and pass them through the filter-graph.

    // assume we have an AVFrame *videoFrame with valid props and buffer
    int ret = -1;
    int pts = videoFrame->pts; // i need to store the pts before running it through the filter.

    avio_seek(imgFmtCtx_->pb, 0,0); // rewind the read head 
    ret = av_read_frame(imgFmtCtx_,imgPkt_);
    ret = avcodec_send_packet(imgDecCtx_, imgPkt_);
    ret = avcodec_receive_frame(imgDecCtx_, imgFrame_);
   
    /** cheap hack to synchronize the timestamps of videoFrame and imgFrame_. We set their time_bases to be equal. 
     * there might a smarter way to rescale their native timestamps and sync them. but this works for now.
    */
    imgFrame_->pts = count_;
    videoFrame->pts = count_;
    imgFrame_->pkt_dts = count_;
    videoFrame->pkt_dts = count_;
    count_++;

    ret = av_buffersrc_add_frame_flags(bufferSrc0Ctx_, videoFrame, AV_BUFFERSRC_FLAG_KEEP_REF);
    ret = av_buffersrc_add_frame_flags(bufferSrc1Ctx_, imgFrame_, AV_BUFFERSRC_FLAG_KEEP_REF); // MEMORY LEAK - APPARENTLY I'M ADDING IT MORE THAN ONCE?
    ret = av_buffersink_get_frame(bufferSinkCtx_, oFrame_);
    av_frame_unref(imgFrame_);

    // restore original pts of videoFrame 
    oFrame_->pts =pts;
    oFrame_->pkt_dts =pts;

Here I've used YUVA420 as the base format when overlaying an RGBA png image on a RGB video.

Answer 2

First, you need the image in the same raw format as the AVFrame::format. Then you can patch the image anywhere on the AVFrame. It will be also useful if the "image" has an alpha channel for transparency. Otherwise, you may resort to color keying.