Merge two Mat-Objects in OpenCV

Question

I wrote a little function in order to be able to "stick" the pixels of an image on top of another, but it somehow doesnt work: While the "shapes" of my "sticked" image is right, the colors are not.

The example flower is the first image, and as a second image I had a black trapezoid png. As you can see, there are multiple problems: 1. The colors are shown weirdly. Actually there are no colors, just greyscale, and some weird stripes as overlay. 2. Alpha values are not respected. The white part of the overlay image is transparent in the png.

Here is my code:

void mergeMats(Mat mat1, Mat mat2, int x, int y){
    //unsigned char * pixelPtr = (unsigned char *)mat2.data;
    //unsigned char * pixelPtr2 = (unsigned char *)mat1.data;
    //int cn = mat2.channels();
    //int cn2 = mat2.channels();
    //Scalar_<unsigned char> bgrPixel;

    for (int i = 0; i < mat2.cols; i++){
        for (int j = 0; j < mat2.rows; j++){
            if (x + i < mat1.cols && y + j < mat1.rows){

                Vec3b &intensity = mat1.at<Vec3b>(j+y, i+x);
                Vec3b intensity2 = mat2.at<Vec3b>(j, i);
                for (int k = 0; k < mat1.channels(); k++) {
                    intensity.val[k] = saturate_cast<uchar>(intensity2.val[k]);
                }
                //pixelPtr2[(i + x)*mat1.cols*cn2 + (j + y)*cn2 + 0] = pixelPtr[(i + x)*mat2.cols*cn + (j + y)*cn + 0];
                //pixelPtr2[(i + x)*mat1.cols*cn2 + (j + y)*cn2 + 1] = pixelPtr[(i + x)*mat2.cols*cn + (j + y)*cn + 1];
                //pixelPtr2[(i + x)*mat1.cols*cn2 + (j + y)*cn2 + 2] = pixelPtr[(i + x)*mat2.cols*cn + (j + y)*cn + 2];
            }
        }
    }
}

The commented code was another approach, but had the same results. So, here are my questions: 1. How do I solve the 2 problems (1. the colors..., 2. alpha ...) 2. How is the pixel-array of any Mat-Object actually, well, organized? I guess it would be easier for me to manipulate those arrays if I knew whats in them.

Answer 1

Because you are iterating mat2 with wrong type. Change Vec3b intensity2 = mat2.at<Vec3b>(j, i); to:

Vec4b intensity2 = mat2.at<Vec4b>(j, i);

and the weird stripes are eliminated. And use intensity2[3] to deal with the alpha channel.

---

Assume that you are reading the black trapezoid png file using -1 flag:

auto trapezoidImg = cv::imread("trapezoid.png", -1);

where the -1 flag specifies that alpha channel is read. Then trapezoidImg is organized in the following format:

[B, G, R, A, B, G, R, A, ......;
  B, G, R, A, B, G, R, A, ......;
  ......
  B, G, R, A, B, G, R, A, ......]

You can print out trapezoidImg, for example using std::cout, to figure out this format.

If you read trapezoidImg using at<Vec3b>, what you get in fact is (B, G, R), (A, B, G), (R, A, B), ......, and this is where the weird stripes came from. Therefore, use at<Vec4b> to read the (R, G, B, A) intensity correctly.

Next, you should define what to do with the alpha channel. You can mixing two Mat or override another, whatever. One simple method is to override mat1 only when the alpha channel in mat2 is large enough:

cv::Vec3b &intensity = mat1.at<cv::Vec3b>(j + y, i + x);
cv::Vec4b intensity2 = mat2.at<cv::Vec4b>(j, i);
for (int k = 0; k < mat1.channels(); k++) {
    if (intensity2.val[3] > 250){  //3 for alpha channel
        intensity.val[k] = cv::saturate_cast<uchar>(intensity2.val[k]);
    }
}

This is enough to deal with your black trapezoid png with transparent background. Or further extend the rule by mixing two Mat:

cv::Vec3b &intensity = mat1.at<cv::Vec3b>(j + y, i + x);
cv::Vec4b intensity2 = mat2.at<cv::Vec4b>(j, i);

auto alphaValue = cv::saturate_cast<uchar>(intensity2.val[3]);
auto alpha = alphaValue / 255.0;

for (int k = 0; k < 2; k++) { //BGR channels only
    intensity.val[k] = cv::saturate_cast<uchar>(
        intensity2.val[k] * alpha + intensity.val[k] * (1.0 - alpha));
}