Assertion Failed Errors at 'copyTo'. Checked image size and type

Question

I have tried searching on this forum, reading the documentation, and googling. I could not find an answer to my question.

Below is a brief of what I am trying to do:

1. The Mat object frame (declared as a private member in the parent class OpenCVVideoWorker) is read from the webcam.
2. A rectangle is drawn on the frame using the position and coordinates given by the Rect_ object ROIbox.
3. The region of interest is thresholded using inRange.
4. The thresholded image in the region of interest is converted to RGB and copied back to the output frame at the same location it was taken out from.

I use Qt for the GUI and Visual Studio as my development environment. When I run the program, I get 'Assertion Failed' errors despite making sure that the rows, columns, and type of the images are same while using copyTo. The error is at frameROIProcessed.copyTo(outputFrame(ROIbox)) (I put in console outputs at regular intervals to find this).

Could you help me find where I am going wrong?

void OpenCVVideoWorker::processFrame() {
    frame.copyTo(outputFrame);

    if (ROIbox.width != 0 && ROIbox.height != 0) {
        rectangle(outputFrame, Point(ROIbox.x, ROIbox.y), cvPoint(ROIbox.x + ROIbox.width, ROIbox.y + ROIbox.height), Scalar(0, 0, 255), 1);
    }

    Mat frameROI = frame(ROIbox);

    if (frameROI.rows != 0 && frameROI.cols != 0) {
        Mat frameROIProcessed;
        cvtColor(frameROI, frameROIProcessed, COLOR_BGR2HSV);
        inRange(frameROIProcessed, Scalar(hueLow, saturationLow, brightnessLow), Scalar(hueHigh, saturationHigh, brightnessHigh), frameROIProcessed);
        cvtColor(frameROIProcessed, frameROIProcessed, COLOR_GRAY2BGR);

        qInfo("1) outputFrame(ROIbox)");
        qInfo(getImageType(outputFrame(ROIbox)).c_str());
        qInfo("2) frameROIProcessed");
        qInfo(getImageType(frameROIProcessed).c_str());
        qInfo("\n");

        frameROIProcessed.copyTo(outputFrame(ROIbox));
    }
    cvtColor(outputFrame, outputFrame, COLOR_BGR2RGB);
}

It is called like this in a function that runs every 16ms in a thread:

void OpenCVVideoWorker::receiveTimerTick() {    
    // Capture video from the webcam
    videoCapture.read(frame);
    if (frame.empty()) {
        qInfo("ERROR! blank frame grabbed");
        return;
    }

    // Process the captured frame
    processFrame();

    // Send the processed frame
    QImage outputImage((const unsigned char *) outputFrame.data, outputFrame.cols, outputFrame.rows, outputFrame.step, QImage::Format_RGB888);
    emit sendImage(outputImage);
}

I get the following errors at the line frameROIProcessed.copyTo(outputFrame(ROIbox)):

Initializing the worker in its constructor...
Opening camera...
1) outputFrame(ROIbox)
8UC3 rows:4 cols:5
2) frameROIProcessed
8UC3 rows:4 cols:5


1) outputFrame(ROIbox)
8UC3 rows:27 cols:22
2) frameROIProcessed
8UC3 rows:27 cols:22


OpenCV(4.0.0-pre) Error: Assertion failed (!fixedSize() || ((Mat*)obj)->size.operator()() == Size(_cols, _rows)) in cv::debug_build_guard::_OutputArray::create, file d:\applications\opencv\source\opencv-master\modules\core\src\matrix_wrap.cpp, line 1227
OpenCV: terminate handler is called! The last OpenCV error is:
OpenCV(4.0.0-pre) Error: Assertion failed (!fixedSize() || ((Mat*)obj)->size.operator()() == Size(_cols, _rows)) in cv::debug_build_guard::_OutputArray::create, file d:\applications\opencv\source\opencv-master\modules\core\src\matrix_wrap.cpp, line 1227

Update: I tried the same code in a non-multithreaded format and it worked. Now I am not sure why the function is not thread safe even if I do the below (I still get the same error, but I think I made sure that only one instance of the function is running at one time):

    if (isFrameBeingProcessed) {
        return;
    }
    isFrameBeingProcessed = true;
    frame.copyTo(outputFrame);

    if (ROIbox.width != 0 && ROIbox.height != 0) {
        rectangle(outputFrame, Point(ROIbox.x, ROIbox.y), cvPoint(ROIbox.x + ROIbox.width, ROIbox.y + ROIbox.height), Scalar(0, 0, 255), 1);
    }

    Mat frameROI = frame(ROIbox);

    if (frameROI.rows != 0 && frameROI.cols != 0) {
        Mat frameROIProcessed;
        cvtColor(frameROI, frameROIProcessed, COLOR_BGR2HSV);
        inRange(frameROIProcessed, Scalar(hueLow, saturationLow, brightnessLow), Scalar(hueHigh, saturationHigh, brightnessHigh), frameROIProcessed);
        cvtColor(frameROIProcessed, frameROIProcessed, COLOR_GRAY2BGR);

        qInfo("1) outputFrame(ROIbox)");
        qInfo(getImageType(outputFrame(ROIbox)).c_str());
        qInfo("2) frameROIProcessed");
        qInfo(getImageType(frameROIProcessed).c_str());
        qInfo((string("Box x: ")+to_string(ROIbox.x) + string(", y: ") + to_string(ROIbox.y) + string(", width: ") + to_string(ROIbox.width) + string(", height: ") + to_string(ROIbox.height)).c_str());
        qInfo((string("Mat width: ") + to_string(outputFrame(ROIbox).cols) + string(", height: ") + to_string(outputFrame(ROIbox).rows)).c_str());
        qInfo("\n");
        frameROIProcessed.copyTo(outputFrame(ROIbox));
    }
    cvtColor(outputFrame, outputFrame, COLOR_BGR2RGB);
    isFrameBeingProcessed = false;

Answer 1

I displayed debugging statements on the console at every step and found what the problem was. The problem was that ROIbox was not thread-safe (thanks @zindarod and @Micka). The GUI which gives the region of interest box ROIbox is on a different thread and ROIbox changes size during the processing of the frame. The simple fix was to add Rect_<int> box = ROIbox; in the beginning of the function processFrame() and to use this copied box in the function.