CODEWITHSUNDEEP
c++cvideogstreamerimage-resizing
robby987
robby987

Reputation: 827

Changing video format on GStreamer

I'm trying to overlay a .png image (with an alpha channel) on gstreamer-1.0 for an application I am writing, however, after a lot of searching the web and reading the documentation I'm still somewhat confused with the method to use.

The method I am using is:

filesrc -> pngdec -> imagefreeze -> videoconvert -> videoscale ->videomixer -> output_of_bin input_of_bin -> videoconvert -> videoscale ->|

This is a small section of a larger application.

My issue is setting the properties on the videoscale and videoconvert elements, I think you're supposed to use GSTCaps, however this does not work. I've tried many ways. This is a member function in a class that provides the overlay. I apologise in advanced for the messiness of the code, but it is experimental.

bool ImageOverlay::CreateElement()
{
  GstPad *source_pad, *sink_pad;
  GstElement *file_source, *png_decoder, *freeze, *colorspace,
      *video_mixer, *identity, * colorspace2, *video_scale_1, *video_scale_2;

  /* Create gstreamer elements */
  _image_overlay_element = gst_bin_new("image-player");

  if(!_image_overlay_element)
  {
    g_printerr ("Pipeline could not be created. Exiting.\n");
    return false;
  }

  // Create file source
  file_source = gst_element_factory_make ("multifilesrc",  "file-source");
  g_object_set (G_OBJECT (file_source), "location", "../Template1.png", NULL);

  if(!file_source)
  {
    g_printerr ("File could not be created. Exiting.\n");
    return false;
  }

  // Decode file
  png_decoder  = gst_element_factory_make ("pngdec", "png-decoder");
  if(!png_decoder)
  {
    g_printerr ("png-decoder could not be created. Exiting.\n");
    return false;
  }

  // Create colour space convertor
  colorspace = gst_element_factory_make("videoconvert", "colorspace");
  if(!colorspace)
  {
    g_printerr ("Colorspace could not be created. Exiting.\n");
    return false;
  }

  // Create colour space convertor
  colorspace2 = gst_element_factory_make("videoconvert", "colorspace2");
  if(!colorspace)
  {
    g_printerr ("Colorspace could not be created. Exiting.\n");
    return false;
  }

  // Create idenentiy component, does nothing but is useful for debug
  identity = gst_element_factory_make("identity", "identity");
  if(!identity)
  {
    g_printerr ("Colorspace could not be created. Exiting.\n");
    return false;
  }

  // Create a video mixer
  video_mixer = gst_element_factory_make("videomixer", "mixer");
  if(!video_mixer)
  {
    g_printerr ("videomixer could not be created. Exiting.\n");
    return false;
  }

  // Create a freeze element
  freeze = gst_element_factory_make("imagefreeze", "freeze");
  if(!freeze)
  {
    g_printerr ("ImageFreeze could not be created. Exiting.\n");
    return false;
  }

  video_scale_1 = gst_element_factory_make("videoscale", NULL);
  if(!video_scale_1)
  {
    g_printerr ("video_scale_1 could not be created. Exiting.\n");
    return false;
  }

  video_scale_2 = gst_element_factory_make("videoscale", NULL);
  if(!video_scale_2)
  {
    g_printerr ("video_scale_2 could not be created. Exiting.\n");
    return false;
  }

  // Add to the bin
  gst_bin_add_many (GST_BIN (_image_overlay_element),
                    file_source,
                    png_decoder,
                    colorspace,
                    colorspace2,
                    freeze,
                    video_mixer,
                    identity,
                    video_scale_1,
                    video_scale_2,
                    NULL);


  if (!gst_element_link_many (file_source, png_decoder, video_scale_1, freeze, NULL))
  {
    std::cout << "Could not link" << std::endl;
  }

  GstCaps * caps = gst_caps_new_simple ("video/x-raw",
                                        "format", G_TYPE_STRING, "ARGB",
                                        NULL);


  GstCaps * caps2 = gst_caps_new_simple("width", G_TYPE_INT, 640,
                                        "height", G_TYPE_INT, 480,
                                        NULL);

  gst_element_link_filtered(video_scale_1, freeze, caps2);
  gst_element_link_filtered(video_scale_2, freeze, caps2);

  if (!gst_element_link_many(freeze, video_scale_2, colorspace, NULL))
  {
    std::cout << "Could not link" << std::endl;
  }

  // Link pads to video mixer
  GstPad * sink_pad_1 = gst_element_get_request_pad (video_mixer, "sink_%u");
  GstPad * sink_pad_2 = gst_element_get_request_pad (video_mixer, "sink_%u");
  GstPad * source_2 = gst_element_get_static_pad (identity, "src");
  GstPad * source_1 = gst_element_get_static_pad (colorspace, "src");

  gst_pad_set_caps(source_2, caps);
  gst_pad_set_caps(source_1, caps);
  gst_caps_unref(caps);

  if (!gst_pad_link ( source_1, sink_pad_1) == GST_PAD_LINK_OK)
  {
    std::cout << "Could not link pads" << std::endl;
    return false;
  }

  if (!gst_pad_link ( source_2, sink_pad_2) == GST_PAD_LINK_OK)
  {
    std::cout << "Could not link pads" << std::endl;
    return false;
  }

  gst_pad_set_active(source_1, true);
  gst_pad_set_active(sink_pad_1, true);
  gst_pad_set_active(source_2, true);
  gst_pad_set_active(sink_pad_2, true);

  // Add pad to this element
  source_pad = gst_element_get_static_pad (video_mixer, "src");
  sink_pad = gst_element_get_static_pad(identity, "sink");
  gst_element_add_pad (_image_overlay_element, gst_ghost_pad_new ("src", source_pad));
  gst_element_add_pad (_image_overlay_element, gst_ghost_pad_new ("sink", sink_pad));

  return true;
}

I would like to take both sources resize them to the same size and overlay the png image on the video. The block mainly works, but I cannot get the resize or alpha channel stuff to work.

Thanks,

Edit:

I have now modified my code and have a slightly better understanding however, it seems that the problem is that that the capfilter will not link if it uses RGBA some other formats do work but are no use. I have no idea why.

bool ImageOverlay::MakeElements()
{
  try
  {
    // Creates the bin at this level
    CreatePipeline();
    GstElement *file_source, *png_decoder, *freeze, *colorspace,
        *video_mixer, *identity, * colorspace2, *video_scale_1, *video_scale_2, *caps_filter1, *caps_filter2;

    // Create elements
    file_source = CreateElementAndAddToBin("multifilesrc",  "file-source");
    png_decoder   = CreateElementAndAddToBin("pngdec", "png-decoder");
    colorspace    = CreateElementAndAddToBin("videoconvert", "colorspace");
    colorspace2   = CreateElementAndAddToBin("videoconvert", "colorspace2");
    identity      = CreateElementAndAddToBin("identity", "identity");
    video_mixer   = CreateElementAndAddToBin("videomixer", "mixer");
    freeze        = CreateElementAndAddToBin("imagefreeze", "freeze");
    video_scale_1 = CreateElementAndAddToBin("videoscale", "scale1");
    video_scale_2 = CreateElementAndAddToBin("videoscale", "scale2");
    caps_filter1  = CreateElementAndAddToBin("capsfilter", "caps_filter1");
    caps_filter2  = CreateElementAndAddToBin("capsfilter", "caps_filter2");

    // Set input file
    g_object_set (G_OBJECT (file_source), "location", "../Template1.png", NULL);

    // Create caps
    GstCaps *caps = gst_caps_new_simple ("video/x-raw",
                                         "format", G_TYPE_STRING, "YV12", //<<< IF THIS IS SET TO ARGB (THE FORMAT I WANT IT FAILS ON LINKING)
                                         "framerate", GST_TYPE_FRACTION, 25, 1,
                                         "pixel-aspect-ratio", GST_TYPE_FRACTION, 1, 1,
                                         "width", G_TYPE_INT, 640,
                                         "height", G_TYPE_INT, 480,
                                         NULL);


    // Set caps on cap filter
    g_object_set(G_OBJECT(caps_filter2), "caps", caps, NULL);
    g_object_set(G_OBJECT(caps_filter1), "caps", caps, NULL);


    // Link up elements
    if (!gst_element_link_many (identity, video_scale_2, colorspace2, caps_filter2, video_mixer, NULL))
    {
      std::cout << "Could not link input branch" << std::endl;  //<<<< IF ARGB OR OTHER FORMATS SUCH AS RGBA ARE USED
    }                                                           // IN CAP FILTER, THIS FAILS, SAME WITH BELOW. IT WORKS WITH YV12!

    if (!gst_element_link_many (file_source, png_decoder, freeze, video_scale_1, colorspace, caps_filter1, video_mixer, NULL))
    {
      std::cout << "Could not link image source branch" << std::endl;
    }

    // Add ghost pads to this bin, allowing it to be used as an element
    SetInputElement(identity);
    SetOutputElement(video_mixer);
  }
  catch(...)
  {
    std::cout << "Overlay generator failed to build, exception thrown" << std::endl;

    // TODO - Clear up in here
    return false;
  }

  return true;
}

Upvotes: 2

Views: 8725

Answers (1)

Mathieu_Du
Mathieu_Du

Reputation: 827

As of gstreamer 1.2, videoconversion is assumed by the mixer, it will automatically select an alpha format if one of the sink pads has alpha.

If you can update to 1.2, the pipeline you need to use will have that form:

gst-launch-1.0 uridecodebin uri=file:///home/meh/Pictures/pitivi/nicetext.png ! imagefreeze ! videomixer name=m sink_0::zorder=0 ! videoconvert ! autovideosink videotestsrc ! videoscale ! video/x-raw, width=640, height=400 ! m.

where nicetext.png is a (png) 640 x 400 image file, with a transparent background and some random text.

The part after the videoscale is a capsfilter, you can create it with:

gst_element_factory_make("capsfilter", "random_name")

then set its caps property with the width and height you require. If this is not known in advance, you will need to setup a pad probe on the pad from which you need to copy the width and height, intercept the caps event and set the caps on the capsfilter at that moment.

If for some reason you can't use 1.2, then you will need a videoconvert and a capsfilter on both branches, and specify the format you require.

Upvotes: 3

Related Questions