Opencv Omnidirectional Calibration Rectification

Question

I have a wide angle lens (>160 degrees). I have tried both the Pinhole and Fisheye models and they seem to provide clear rectified images with a little FOV sacrifice. So I decided to give a try with the Omnidirectional Model.

With the Omnidirectional model:

double rms = cv::omnidir::stereoCalibrate(Omni_L.RealP, Omni_L.ImageP, Omni_R.ImageP, Omni_L.img.size(), Omni_R.img.size(), K1, xi1, D1, K2, xi2, D2, rvec, tvec, rvecsL, tvecsL, flags, critia);
std::cout << "RMS : " << rms << std::endl;

I Get the following results after the Stereo-calibration

RMS : 0.527522

 Camera_Matrix1 :
 [773.9380049495828, 0, 394.6697338356358;
 0, 776.2094223216956, 382.016762545214;
 0, 0, 1]

 Xi-1 :[1.804945374650817]

 Distortion Parameters1:
 [0, 0, -0.009983732008104793, -0.004530718062523464]

 Camera_Matrix2:
 [783.1807043871861, 0, 393.2120687911561;
 0, 784.4790955477508, 386.7954078861521;
 0, 0, 1]

 Xi-2 :[1.837909340970556]

 Distortion Parameters2:
 [0, 0, -0.009779422171305124, -0.003723193186299092]

 rvec :[-0.003821627764900316;
 0.005567569400892289;
 0.001491183114878044]

 tvec:[-63.09243360480385;
 0.1195353275271446;
 0.9082801155496641]

 R: 
 [1, 0, 0;
 0, 1, 0;
 0, 0, 1]

 P: 
 [773.9380049495828, 0, 394.6697338356358;
 0, 776.2094223216956, 382.016762545214;
 0, 0, 1]

Then I do undistortion:

    cv::Mat R = cv::Mat::eye(3, 3, CV_32FC1);
    cv::Mat Mapx, Mapy;
    cv::Mat P(3, 3, CV_32FC1);
    P = K1;
    
    cv::Mat orid = cv::imread("Left\\1.jpg");
    std::cout << orid.size();
    std::cout << "R : "<<R<<std::endl;
    std::cout << "P : " << P << std::endl;
    cv::Size s= orid.size();
    try {
        cv::omnidir::initUndistortRectifyMap(K1, D1, xi1, R, P, s, CV_32FC1, Mapx, Mapy, cv::omnidir::RECTIFY_PERSPECTIVE);// Knew, new_size);
        cv::remap(orid, DC, Mapx, Mapy, cv::INTER_CUBIC);
    }
    catch (cv::Exception & e)
    {
        std::cerr << e.msg << std::endl; // output exception message
    }
    std::string Save_Original = "Distorted_Original" + std::to_string(10) + EXT;
    cv::imwrite(Save_Original, orid);
    std::string Save_Corrected = "Distorted_Corrected" + std::to_string(10) + EXT;
    cv::imwrite(Save_Corrected, DC);

Question 1:

Why the Radial distortion is zero?

Question 2:

The output of the undistortion image with rectification drastically reduces the FOV. Is there something wrong in my code?

Original Image: Distortion Corrected Image:

Question 3:

What does the P and R do in the cv::omnidir::initUndistortRectifyMap() function?

Question 4:

Need Ideas on how to get good rectified images with more FoV with omnidirectional calibration?

---

Post Trials:

1. suggested by @sushi(commented below) to try RECTIFY_CYLINDRICAL for the same image and below is the result:

   cv::omnidir::RECTIFY_CYLINDRICAL -> Results are worse

Answer 1

First I recommend you to read this tutorial: https://docs.opencv.org/master/dd/d12/tutorial_omnidir_calib_main.html Answering your questions:

1. Distortion coefficients depend on the distortion model that you choose.
2. Look at the tutorial (image rectification part), if you want to preserve all fov you have to change the flag to RECTIFY_CYLINDRICAL
3. P is "new" camera matrix for image after rectification, R is rotation transform between the original and object space. You can skip both of these parameters if you undistort your images using cv::omnidir::undistortImage. Look in the tutorial about recommended values for knew.
4. Look at the answer in point 2.

This is the result of my approach using RECTIFY_PERSPECTIVE, I don't know why RECTIFY_CYLINDRICAL does not work in this case: To achieve it simply undistort your image like this:

cv::Size s = orid.size();
cv::Mat Knew = cv::Mat(cv::Matx33f(s.width / 4, 0, s.width / 2,
        0, s.height / 4, s.width / 2,
        0, 0, 1));
cv::MAt undistorted;
cv::omnidir::undistortImage(orid, undistorted, K1, D1, xi1, cv::omnidir::RECTIFY_PERSPECTIVE, Knew, orid.size());