Move GTK+ window in real-time

Question

I am creating a moving window that uses face detection coordinates as an input to assign the window's new position. Currently, the face detection is functional, but the window does not get displayed until the very end of the capture loop.

My questions are:
-How can I keep the window in view the entire time the image capture and face detection is taking place?
-Is a "gtk_main" loop necessary, and is it being used properly in this scenario?
-Why does the window not open even when "gtk_widget_show (window)" is placed in the capture loop?
-Is there a better forum for more detailed GTK+ question?

I would like to model this after OpenCV's "moveWindow" function. This function works perfectly for what I need, the only problem with using this function is that I am not able to customize the window.

Source code for OpenCV's "moveWindow" function: Look under window.cpp and window_gtk.cpp https://github.com/opencv/opencv/tree/master/modules/highgui/src

#include "FlyCapture2.h"
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/objdetect/objdetect.hpp>
#include <opencv2/core/cuda.hpp>
#include <opencv2/cudalegacy/NCVHaarObjectDetection.hpp>
#include <opencv2/cudaobjdetect.hpp>
#include <math.h>
#include <thread>
#include <iostream>
#include <vector>
#include <gtk-3.0/gtk/gtk.h>

using namespace FlyCapture2;


cv::Ptr<cv::cuda::CascadeClassifier> face_detect;

int x,y;

void detect_faces(cv::Mat img, cv::cuda::GpuMat buf)
{
    std::vector<cv::Rect>faces;

    //Detect faces
    ...

    if (faces.size() > 0) 
    {
        float x_f = faces[0].x;
        float y_f = faces[0].y;
        x = roundf(x_f*40/51);
        y = roundf(y_f*135/256);    

    }

}

int main( int   argc, char *argv[])
{

    //Camera initialization
    ...

    //face detect variables
    face_detect = cv::cuda::CascadeClassifier::create("/home/nvidia/opencv/data/haarcascades_cuda/haarcascade_frontalface_default.xml");
    cv::cuda::GpuMat objbuf;

    //GTK+ Params
    GtkWidget *window;
    gtk_init (&argc, &argv);
    window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    gtk_window_set_decorated(GTK_WINDOW (window),FALSE);
    gtk_window_set_position(GTK_WINDOW (window), GTK_WIN_POS_CENTER);
    gtk_widget_show  (window);

    // capture loop
    double t = (double)cv::getTickCount();
    for (int i=0;i<100;i++)
    {
        // Get the image
        ...

        // convert to OpenCV Mat
        ...


        //Detect Faces
        detect_faces(image,objbuf);
            std::cout<<"x: "<<x<<" "<<"y: "<<y<<std::endl;
        gtk_window_move(GTK_WINDOW (window),x,y);
        while (gtk_events_pending())
            gtk_main_iteration ();

    }

    //Record Time
    t = ((double)cv::getTickCount() - t)/cv::getTickFrequency();
        std::cout << "Time: " << (t/100)*1000 << std::endl;

    //Disconnect Camera
    camera.StopCapture();
    camera.Disconnect();

    gtk_main();

    return 0;
}

Answer 1

The best approach is to separate the face recognition procedure and the GUI operations in two different threads, the GUI should always run in the main thread (or the one that created the window in the first place, this is not strictly needed on X11 but it is in Win32 and Cocoa GTK backends for instance).

The recognition thread can then busy loop as needed and 'send' the window updates to the main thread using and idle callback. This is the commonly used GTK approach to multithread.

Here is some code (a support function and an alternative capture loop) that explains the approach:

/// support function
struct WindowData
{
   GtkWindow win;
   int x, y;
};

int move_window(WindowData *p)
{
    gtk_move_window(p->win, p->x, p->y);
    delete p;
    return FALSE;
}

[...]

// updated capture loop inside main (capture the variables you need, or this if you are working in a class environment
std::thread t([window, image, objectbuf]{
  for (int i=0;i<100;i++) {
      // Get the image
      ...
      // convert to OpenCV Mat
      ...

      //Detect Faces
      detect_faces(image,objbuf);
      WindowData *p = new WindowData();
      p.win = window;
      p.x = x; p.y = y;
      g_idle_add((GFunction)move_window, p);

    }
});
gtk_main();
t.join();

[...]

Note that you can also make "window" a global variable (or a class member) and make x and y std::atomic to avoid the need to allocate/deallocate WindowData for every window movement.