Opencv image to array issue

Question

Im having an issue using opencv trying to convert an image to an array. The conversion works however i seem to have incorrect dimensions in the resulting array:

#include <opencv2/opencv.hpp>

int main()
{
    auto img = cv::imread("test.jpg", CV_LOAD_IMAGE_COLOR);
    std::cout << "img cols: " << img.cols << " img rows: "
        << img.rows << " channels: " << img.channels() << std::endl;

    std::vector<float> array2;
    if (img.isContinuous()) {
        array2.assign((float*)img.ptr(0), (float*)(img.ptr(img.rows - 1)) + img.cols);
        std::cout << array2.size() << "\n";
    }

    return 0;
}

The output from the first print line results in :

img cols: 416 img rows: 416 channels: 3

Which is correct, however after assigning the data to the array the dimensions are : 518336 , when they should be 519168 (416*416*3).

Could anyone possibly suggest what exactly is causing the resulting array to be smaller than expected?

Answer 1

There are several problems with your code:

• First of all, cv::imread("test.jpg", CV_LOAD_IMAGE_COLOR); will (on success) return a cv::Mat with datatype CV_8UC3, however you're accessing the elements as floats. This means that the values you will read will be garbage, and you will also end up reading past the end of the pixel buffer.

  If you want floats, then you need to do some conversion/casting, either before or during the act of copying.

• The second problem lies in your calculation of the "end" pointer, where you seem to forget that you're dealing with a multi-channel cv::Mat. In case of a CV_8UC3 matrix, each pixel is represented by 3 bytes, hence there are cols*channels bytes per row. (That's why you're short by 2*416 elements)

• Not really a problem, but a limitation -- your code only works for continuous Mats.

---

I would take a somewhat different approach, and take advantage of functionality provided by OpenCV.

Option 1

Use cv::Mat::copyTo, since OutputArray can wrap a std::vector<T>. However, for this to work, the source Mat needs to have 1 channel and 1 row. We can achieve this efficiently using cv::Mat::reshape, but the Mat needs to be continuous, so that limitation stays.

std::vector<uchar> to_array_v1(cv::Mat3b const& img)
{
    std::vector<uchar> a;
    if (img.isContinuous()) {
        img.reshape(1, 1).copyTo(a);
    }
    return a;
}

Option 2

Use MatIterators which we can get using cv::Mat::begin and cv::Mat::end. The iterators will work correctly even on a non-continuous Mat, however we need them to iterate over bytes, so we need to reshape the matrix to a single channel one. Since we're not changing the number of rows, the reshape will also work on a non-continuous Mat.

std::vector<uchar> to_array_v2(cv::Mat3b const& img)
{
    cv::Mat1b tmp(img.reshape(1));
    return std::vector<uchar>(tmp.begin(), tmp.end());
}

Option 3

The approach suggested by Silencer, using the rather poorly documented cv::Mat::datastart and cv::Mat::dataend members. The documentation of cv::Mat::locateROI sheds some more light on the meaning of those member variables:

However, each submatrix contains information (represented by datastart and dataend fields) that helps reconstruct the original matrix size and the position of the extracted submatrix within the original matrix.

This means that this approach has 2 limitations: it needs a continous matrix, and it won't work correctly for a submatrix, even if it's continuous. (Specifically, for a continuous submatrix, it would return the entire buffer of the "parent" matrix)

std::vector<uchar> to_array_v3(cv::Mat3b const& img)
{
    std::vector<uchar> a;
    if (img.isContinuous() && !img.isSubmatrix()) {
        a.assign(img.datastart, img.dataend);
    }
    return a;
}

---

Test Code

#include <opencv2/opencv.hpp>

#include <iostream>
#include <numeric>
#include <vector>


// Paste implementations from the answer here


cv::Mat3b test_image()
{
    cv::Mat1b m(4, 4);
    std::iota(m.begin(), m.end(), 0);

    cv::Mat3b img;
    cv::merge(std::vector<cv::Mat1b>{ m * 3, m * 3 + 1, m * 3 + 2 }, img);
    return img;
}

void print(cv::Mat3b const& img)
{
    std::cout << "Continuous: " << (img.isContinuous() ? "yes" : "no") << '\n';
    std::cout << "Submatrix: " << (img.isSubmatrix() ? "yes" : "no") << '\n';
    std::cout << img << "\n";
}

void print(std::vector<uchar> const& a)
{
    if (a.empty()) {
        std::cout << "empty";
    } else {
        for (auto n : a) {
            std::cout << int(n) << ' ';
        }
    }
    std::cout << "\n";
}

void test(cv::Mat3b const& img)
{
    print(img);
    print(to_array_v1(img));
    print(to_array_v2(img));
    print(to_array_v3(img));
}

int main()
{
    cv::Mat3b img(test_image());
    test(img);

    cv::Mat3b img2(img(cv::Rect(0, 0, 3, 3)));
    test(img2);

    cv::Mat3b img3(img(cv::Rect(1, 1, 3, 1)));
    test(img3);

    return 0;
}

---

Running this program will produce the following output:

Continuous: yes
Submatrix: no
[  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11;
  12,  13,  14,  15,  16,  17,  18,  19,  20,  21,  22,  23;
  24,  25,  26,  27,  28,  29,  30,  31,  32,  33,  34,  35;
  36,  37,  38,  39,  40,  41,  42,  43,  44,  45,  46,  47]
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 
Continuous: no
Submatrix: yes
[  0,   1,   2,   3,   4,   5,   6,   7,   8;
  12,  13,  14,  15,  16,  17,  18,  19,  20;
  24,  25,  26,  27,  28,  29,  30,  31,  32]
empty
0 1 2 3 4 5 6 7 8 12 13 14 15 16 17 18 19 20 24 25 26 27 28 29 30 31 32 
empty
Continuous: yes
Submatrix: yes
[ 15,  16,  17,  18,  19,  20,  21,  22,  23]
15 16 17 18 19 20 21 22 23 
15 16 17 18 19 20 21 22 23 
empty

Answer 2

Mat img = imread("test.png");
std::vector<uchar> arr;

// convert Mat of CV_8UC3 to std::vector<uchar> if continuous
if(img.isContinuous()){
    arr.assign(img.datastart, img.dataend);
}