How to fix unhandled OpenCVException from calling Cv2.SolvePnP()?

Question

I try to call Cv2.SolvePnP() of opencvsharp like so:

using System.Collections.Generic;

using OpenCvSharp;

namespace Test
{
    public class SolvePnPTest
    {
        public static void Main(string[] args)
        {
            var objectPoints = new List<Point3f>();
            objectPoints.Add(new Point3f(1, 2, 3));
            objectPoints.Add(new Point3f(1, 2, 3));
            objectPoints.Add(new Point3f(1, 2, 3));
            objectPoints.Add(new Point3f(1, 2, 3));

            var imagePoints = new List<Point2f>();
            imagePoints.Add(new Point2f(1, 2));
            imagePoints.Add(new Point2f(1, 2));
            imagePoints.Add(new Point2f(1, 2));
            imagePoints.Add(new Point2f(1, 2));

            var rvec = new double[3];
            var tvec = new double[3];

            var cameraMatrix = new double[,] { { 1, 2, 3 }, { 4, 5, 6 }, { 7, 8, 9 } };
            var distortionCoefficients = new double[] { 1, 2, 3, 4, 5 };

            Cv2.SolvePnP(
                objectPoints,
                imagePoints,
                cameraMatrix,
                distortionCoefficients,
                out rvec,
                out tvec);
        }
    }
}

The call to Cv2.SolvePnP fails with an unhandled OpenCVException:

src.size == dst.size && src.channels() == dst.channels()

What is src? What is dst?

The details of the error including the stacktrace look like that:

HResult=-2146232832
Message=src.size == dst.size && src.channels() == dst.channels()
Source=OpenCvSharp
ErrMsg=src.size == dst.size && src.channels() == dst.channels()
FileName=..\..\..\modules\core\src\convert.cpp
FuncName=cvConvertScale
Line=5475
StackTrace:
     at OpenCvSharp.NativeMethods.<>c.<.cctor>b__1332_0(ErrorCode status, String funcName, String errMsg, String fileName, Int32 line, IntPtr userdata)
     at OpenCvSharp.NativeMethods.calib3d_solvePnP_vector(Point3f[] objectPoints, Int32 objectPointsLength, Point2f[] imagePoints, Int32 imagePointsLength, Double[,] cameraMatrix, Double[] distCoeffs, Int32 distCoeffsLength, Double[] rvec, Double[] tvec, Int32 useExtrinsicGuess, Int32 flags)
     at OpenCvSharp.Cv2.SolvePnP(IEnumerable`1 objectPoints, IEnumerable`1 imagePoints, Double[,] cameraMatrix, IEnumerable`1 distCoeffs, Double[]& rvec, Double[]& tvec, Boolean useExtrinsicGuess, SolvePnPFlags flags)
     at Test.SolvePnPTest.Main(String[] args) in <file path>.
InnerException: 

What's the problem with my code?

The assertion in convert.cpp is in this opencv function:

CV_IMPL void
cvConvertScale( const void* srcarr, void* dstarr,
                double scale, double shift )
{
    cv::Mat src = cv::cvarrToMat(srcarr), dst = cv::cvarrToMat(dstarr);

    CV_Assert( src.size == dst.size && src.channels() == dst.channels() );
    src.convertTo(dst, dst.type(), scale, shift);
}

Answer 1

Well, I think the only reason for the

src.size == dst.size && src.channels() == dst.channels() in cvConvertScale()

is that you're using a function in which the source matrix (src) isn't the same as the destination matrix (dest) [in terms of mat.size or/and mat.channels (one of them has different dimensions, is uninitialized, one is RGB when the other is grayscale ...)] where they have to be the same.

That's the "why ?" for that.

Now, I am no expert with Lists and I'm not sure if this makes sense, but in SolvePnp documentation, they use vectors, get them a try :

 std::vector<Point3f> objectPoints;
 objectPoints.push_back(new Point3f(1, 2, 3));
 objectPoints.push_back(new Point3f(1, 2, 3));
 objectPoints.push_back(new Point3f(1, 2, 3));
 objectPoints.push_back(new Point3f(1, 2, 3));

Answer 2

As this overload of the method fails,

public static void SolvePnP(
    IEnumerable<Point3f> objectPoints,
    IEnumerable<Point2f> imagePoints,
    double[,] cameraMatrix,
    IEnumerable<double> distCoeffs,
    out double[] rvec,
    out double[] tvec,
    bool useExtrinsicGuess = false,
    SolvePnPFlags flags = SolvePnPFlags.Iterative
)

I tried the other

public static void SolvePnP(
    InputArray objectPoints,
    InputArray imagePoints,
    InputArray cameraMatrix,
    InputArray distCoeffs,
    OutputArray rvec,
    OutputArray tvec,
    bool useExtrinsicGuess = false,
    SolvePnPFlags flags = SolvePnPFlags.Iterative
)

which executes. Full example code:

using System.Collections.Generic;

using OpenCvSharp;

namespace Test
{
    public class SolvePnPTest
    {
        public static void Main(string[] args)
        {
            var objectPoints = new List<Point3f>();
            objectPoints.Add(new Point3f(1, 2, 3));
            objectPoints.Add(new Point3f(1, 2, 3));
            objectPoints.Add(new Point3f(1, 2, 3));
            objectPoints.Add(new Point3f(1, 2, 3));

            var imagePoints = new List<Point2f>();
            imagePoints.Add(new Point2f(1, 2));
            imagePoints.Add(new Point2f(1, 2));
            imagePoints.Add(new Point2f(1, 2));
            imagePoints.Add(new Point2f(1, 2));

            var rvec = new double[3];
            var tvec = new double[3];

            var cameraMatrix = new double[,] { { 1, 2, 3 }, { 4, 5, 6 }, { 7, 8, 9 } };
            var distortionCoefficients = new double[] { 1, 2, 3, 4, 5 };

            /*
            Cv2.SolvePnP(
                objectPoints,
                imagePoints,
                cameraMatrix,
                distortionCoefficients,
                out rvec,
                out tvec);
             * 
             * */

            Cv2.SolvePnP(
                new Mat(objectPoints.Count, 3, MatType.CV_64F, objectPoints.ToArray()),
                new Mat(imagePoints.Count, 2, MatType.CV_64F, imagePoints.ToArray()),
                new Mat(3, 3, MatType.CV_64F, cameraMatrix),
                new Mat(5, 1, MatType.CV_64F, distortionCoefficients),
                new Mat(3, 1, MatType.CV_64F),
                new Mat(3, 1, MatType.CV_64F));
        }
    }
}