LiDAR Points to Camera Projection – Only Half of the Points Are Correct After 180° Rotation

Question

Issue: LiDAR Points to Camera Projection – Only Half of the Points Are Correct After 180° Rotation

I am trying to project LiDAR points onto a camera image using OpenCV and Open3D. I followed the algorithm described in the OpenCV camera calibration documentation (link) and used the OpenCalib repository for calibration. However, when visualizing the projected points, only half of the LiDAR points align correctly, while the other half are misaligned or missing.

Setup Details:

LiDAR Model: LSLIDAR C16 (Image of LsLidar Li-DAR coordinates)
Calibration Method: OpenCalib repository
Simulation Test: In Gazebo, this issue was resolved by a simple 180° rotation on the Z-axis, but this does not work in the real-world setup.
Observations:
- The issue only occurs when the LiDAR is turned 180° away from the camera (camera is 180° from the Li-DAR's y-axis, refer image above).
- When the LiDAR’s Y-axis is aligned with the camera, the calibration works correctly.

Steps I Followed:

Calibrated the camera and obtained the intrinsic matrix (K).
Estimated extrinsics (rotation R and translation T) between the camera and LiDAR.
Transformed LiDAR points to the camera frame using:
```
lidar_points_cam = R @ lidar_points + T
```

Applied the intrinsic projection to get pixel coordinates:

pixel_coords = K @ lidar_points_cam
pixel_coords /= pixel_coords[2]  # Normalize by depth

Plotted the projected points on the image using OpenCV.

Issue Observed:

The right half of the LiDAR points align correctly, but the left half is misaligned or missing.
Removing the 180° transformation fixes one side but misaligns the other.
The same calibration method works perfectly in simulation and when the LiDAR’s Y-axis is aligned with the camera.

Questions:

Why does this issue occur only after a 180° LiDAR rotation?
How can I properly adjust the transformation to match the real-world setup?

Note: Can't add actual images of the problem as the device I am working on is still waiting for a patent approval.

Why does this issue occur only after a 180° LiDAR rotation?
Could this be a transformation issue (incorrect R or T) or a calibration limitation?
How can I properly adjust the transformation to match the real-world setup?

Edit:

Part pseudo-code

Function 1: project_velo_to_image(pts_3d_velo) Purpose: Projects 3D LiDAR points into the 2D camera image plane.

Convert Rotation Matrix (R0) to Homogeneous Form

Append a zero row to R0 to make it 4x3. Append a zero column and a [0, 0, 0, 1] row to make it 4x4. Compute the Projection Matrix (p_r0_rt)

Multiply the camera projection matrix P with R0 homogeneous (PxR0). Multiply the result with LiDAR-to-camera transformation matrix V2C (PxROxRT). Convert 3D LiDAR Points to Homogeneous Coordinates

Append an extra column of ones to pts_3d_velo. Project LiDAR Points into Image Plane

Multiply the projection matrix (p_r0_rt) with the homogeneous LiDAR points. Normalize by dividing the x, y pixel coordinates by the depth (Z). Return the 2D projected points (image pixel coordinates).

Function 2: get_lidar_in_image_fov(pc_velo, xmin, ymin, xmax, ymax, clip_distance=0.01) Purpose: Filters LiDAR points to keep only those that fall inside the camera's field of view (FOV).

Project LiDAR Points to Image Plane

Call project_velo_to_image(pc_velo) to get 2D pixel coordinates. Apply Field of View (FOV) Filtering

Keep points where x is within image width (xmin ≤ x < xmax). Keep points where y is within image height (ymin ≤ y < ymax). Apply Distance Filtering

Remove points where X (LiDAR forward direction) is too close (X > clip_distance). Return the Filtered LiDAR Points

Edit 2:

This is the algorithm I'm currently using for the lidar and camera, this worked for simulation and real world (when y axis is aligned with camera)

class LiDAR2Camera(object):
    def __init__(self, calib_file):
        calibs = self.read_calib_file(calib_file)
        P = calibs["Camera_matrix"]
        self.P = np.reshape(P, [3, 4])
        # Rigid transform from Velodyne coord to reference camera coord
        V2C = calibs["Tr_velo_to_cam"]
        self.V2C = np.reshape(V2C, [3, 4])
        # Rotation from reference camera coord to rect camera coord
        R0 = calibs["R0_rect"]
        self.R0 = np.reshape(R0, [3, 3])
 
    def read_calib_file(self, filepath):
        """ Read in a calibration file and parse into a dictionary. """
        data = {}
        with open(filepath, "r") as f:
            for line in f.readlines():
                line = line.rstrip()
                if len(line) == 0:
                    continue
                key, value = line.split(":", 1)
                try:
                    data[key] = np.array([float(x) for x in value.split()])
                except ValueError:
                    pass
        return data
    
    def cart2hom(self, pts_3d):
        """ Input: nx3 points in Cartesian
            Oupput: nx4 points in Homogeneous by pending 1
        """
        n = pts_3d.shape[0]
        pts_3d_hom = np.hstack((pts_3d, np.ones((n, 1))))
        return pts_3d_hom
    
    def project_velo_to_image(self, pts_3d_velo):
        '''
        Input: 3D points in Velodyne Frame [nx3]
        Output: 2D Pixels in Image Frame [nx2]
        '''
        # NORMAL TECHNIQUE
        R0_homo = np.vstack([self.R0, [0, 0, 0]])
        R0_homo_2 = np.column_stack([R0_homo, [0, 0, 0, 1]])
        p_r0 = np.dot(self.P, R0_homo_2) #PxR0
        p_r0_rt = np.dot(p_r0, np.vstack((self.V2C, [0, 0, 0, 1]))) #PxROxRT
        pts_3d_homo = np.column_stack([pts_3d_velo, np.ones((pts_3d_velo.shape[0],1))])
        p_r0_rt_x = np.dot(p_r0_rt, np.transpose(pts_3d_homo))#PxROxRTxX
        pts_2d = np.transpose(p_r0_rt_x)
        
        pts_2d[:, 0] /= pts_2d[:, 2]
        pts_2d[:, 1] /= pts_2d[:, 2]
        return pts_2d[:, 0:2]
    
    def get_lidar_in_image_fov(self, pc_velo, xmin, ymin, xmax, ymax, return_more=False, clip_distance=0.01):
        """ Filter lidar points, keep those in image FOV """
        pts_2d = self.project_velo_to_image(pc_velo)
        fov_inds = (
            (pts_2d[:, 0] < xmax)
            & (pts_2d[:, 0] >= xmin)
            & (pts_2d[:, 1] < ymax)
            & (pts_2d[:, 1] >= ymin)
        )
        fov_inds = fov_inds & (pc_velo[:, 0] > clip_distance)
        imgfov_pc_velo = pc_velo[fov_inds, :]
        if return_more:
            return imgfov_pc_velo, pts_2d, fov_inds
        else:
            return imgfov_pc_velo
    
    def lidar_camera_fusion(self, pred_bboxes, image):
        img_bis = image.copy()
        cmap = plt.cm.get_cmap("hsv", 256)
        cmap = np.array([cmap(i) for i in range(256)])[:, :3] * 255
        distances = []
        bearings = []
        return_distances = []
        return_bearings = []
        
        for box in pred_bboxes:
            distances = []
            bearings = []
            
            for i in range(self.imgfov_pts_2d.shape[0]):
                # Get LiDAR coordinates
                x_coord = self.imgfov_pc_velo[i, 0]  # Forward axis
                y_coord = self.imgfov_pc_velo[i, 1]  # Lateral axis
                depth = math.sqrt(x_coord**2 + y_coord**2)  # True distance
 
                # Check if the point is within the bounding box
                if rectContains(box, self.imgfov_pts_2d[i], image.shape[1], image.shape[0], shrink_factor=0.1):
                    distances.append(depth)
 
                    # Calculate the bearing angle (positive is left, negative is right)
                    bearing = math.degrees(math.atan2(y_coord, x_coord))
                    bearings.append(bearing)
 
                    # Visualization (Optional)
                    color = cmap[min(int(510 / depth), 255), :]
                    cv2.circle(
                        img_bis,
                        (int(np.round(self.imgfov_pts_2d[i, 0])), int(np.round(self.imgfov_pts_2d[i, 1]))),
                        2,
                        color=tuple(color),
                        thickness=-1,
                    )
 
            if len(distances) > 2:
                # Filter outliers and get the best distance
                distances = filter_outliers(distances)
                best_distance = get_best_distance(distances)
 
                # Get the average bearing angle (or use another method)
                best_bearing = statistics.median_high(bearings)
 
                return_distances.append(best_distance)
                return_bearings.append(best_bearing)
            
                # Annotate the image with distance and bearing
                center_x = int(box[0] + (box[2] - box[0]) / 2)
                center_y = int(box[1] + (box[3] - box[1]) / 2)
                cv2.putText(
                    img_bis,
                    f"{round(best_distance, 2)}m, {round(best_bearing, 2)}°",
                    (center_x, center_y),
                    cv2.FONT_HERSHEY_SIMPLEX,
                    0.75,
                    (255, 0, 0),
                    2,
                    cv2.LINE_AA,
                )
 
        return img_bis, return_distances, return_bearings

LiDAR Points to Camera Projection – Only Half of the Points Are Correct After 180° Rotation

Issue: LiDAR Points to Camera Projection – Only Half of the Points Are Correct After 180° Rotation

Setup Details:

Steps I Followed:

Issue Observed:

Questions:

Edit:

Part pseudo-code

This is the algorithm I'm currently using for the lidar and camera, this worked for simulation and real world (when y axis is aligned with camera)

Answers (0)

Related Questions

LiDAR Points to Camera Projection – Only Half of the Points Are Correct After 180&#176; Rotation

Issue: LiDAR Points to Camera Projection – Only Half of the Points Are Correct After 180° Rotation

Setup Details:

Steps I Followed:

Issue Observed:

Questions:

Edit:

Part pseudo-code

This is the algorithm I'm currently using for the lidar and camera, this worked for simulation and real world (when y axis is aligned with camera)

Answers (0)

Related Questions

LiDAR Points to Camera Projection – Only Half of the Points Are Correct After 180° Rotation