Orbit camera around point (from scratch, no libraries)

Question

I'm trying to write a software rasterizer from scratch without using libraries like GLFW, OpenGL, etc. I've made a lot of progress but am currently stuck trying to get the camera to orbit properly around the model. At this point I am not sure if it's a problem with the logic in generating the new camera location or with the underlying math I've implemented (quat/matrix/vector logic).

Example of the orbiting problem

My attempt is outlined below.

Orbit logic

void Camera::Orbit(float DX, float DY)
{
    // Get the camera rotation as a vector
    const TVector3<float> CameraOrientation = (Translation - TargetTranslation).Normalized(); // Target is always [0,0,0]

    // Convert delta X and Y to radians
    const float Pitch = Math::DegreesToRadians(DX); // DX * (P_PI / 180.f)
    const float Yaw = Math::DegreesToRadians(DY); // DY * (P_PI / 180.f)

    // Construct a quaternion with the derived Pitch and Yaw
    TQuat<float> QuatRotation{Pitch, Yaw, 0};
            
    // Construct a quaternion with the derived camera orientation
    const TQuat<float> QuatTranslation(CameraOrientation.X, CameraOrientation.Y, CameraOrientation.Z, 0);
            
    // Multiply Qr * Qt * Qr⁻¹
    const TQuat<float> QuatResult = QuatRotation * QuatTranslation * QuatRotation.GetConjugate();
            
    // Take X/Y/Z components of the result and apply them as the X/Y/Z components of
    // the new rotated orientation
    const TVector3<float> RotatedOrientation{QuatResult.X, QuatResult.Y, QuatResult.Z};

    // Multiply by zoom distance to get the final location
    const TVector3<float> NewTranslation = RotatedOrientation * Zoom;

    // Set new location as the current location
    Translation = NewTranslation;
}

Quaternion class

template <typename T>
struct TQuat
{
    T X;
    T Y;
    T Z;
    T W;

    TQuat(T InX, T InY, T InZ, T InW) : X(InX), Y(InY), Z(InZ), W(InW) {}

    TQuat GetConjugate()
    {
        return {-X, -Y, -Z, W};
    }

    TQuat<T> operator*(const TQuat<T>& Other) const
    {
        T TempW = W * Other.W - X * Other.X - Y * Other.Y - Z * Other.Z;
        T TempX = W * Other.X + X * Other.W + Y * Other.Z - Z * Other.Y;
        T TempY = W * Other.Y - X * Other.Y + Y * Other.W + Z * Other.X;
        T TempZ = W * Other.Z + X * Other.Z - Y * Other.X + Z * Other.W;

        return TQuat<T>{TempX, TempY, TempZ, TempW};
    }
}

Vector class

template <typename T>
struct TVector3
{
    T X;
    T Y;
    T Z;

    TVector3(T InX) : X(InX), Y(InX), Z(InX) {}

    void Normalize()
    {
        const T SquareSum = (X * X) + (Y * Y) + (Z * Z);
        const T Scale = Math::InvSqrt(SquareSum);

        X *= Scale;
        Y *= Scale;
        Z *= Scale;
    }
}

It ends up doing this weird orbit where it's not adhering to moving in a single axis. The red line illustrates the click and drag direction. As far as I can tell, the actual view matrix is correct - if I just move directly on any axis, without respect to direction (like moving forward is (0,0,+1) in world-space), the camera moves as expected.

Ultimately I am just trying to get the camera to orbit around the origin (0,0,0) based on mouse click and drag.