Can the model coordinates and image coordinates be the same when using solvepnp?

Question

Currently, I am trying to implement a camera pose tracking system. I have a set of model coordinate points(3D) of the previous frame and the image coordinates(2D) of the current frame.

I have also set an identity matrix(4x4) as the initial pose of the camera. For each new rvec and tvec calculated, I convert the rvec into a 3x3 Rotaton Matrix using rodrigues and then create a 4x4 homogenous transformation matrix and multiply it with the previous pose.

The rotation seems to be working properly, but the translation vector from the matrix seems to be moving only when rotated and not when translated, i.e it represents rotation.

Could it be that both my model coordinates and image coordinate systems are the same?

Edit: I am attempting to track the camera position using 3D-2D correspondences( Visual Odometry ) with a RGBD camera.

Update: Solved the issue. I seemed to be taking the wrong column of the Transformation matrix