Updating position and rotation of a RigidBody or MeshCollider and making it follow along it's child mesh, using react-three/rapier or cannon

Question

So I have a react-three/fiber scene component, where I am animating a rigged character. I want to apply colliders to eg. the hands, so that the movement and rotation of the skinned mesh will apply to its collider, and thereby not having the collider receive force as such, but merely going where ever the mesh is going. Point being the hand can collide and interact with objects in the scene.

Now, I tried to achieve the same result with react-three/cannon but did not have any luck here as well.

Finally I tried the Three.js add-on, oriented bounding box (OBB), and actually achieve the desired outcome, but this does nothing more than detect a collision and does not provide physics or support hull colliders as Rapier does.

Please, if anyone has ideas how to pull this off, please let me know.

Here is my little snippet with react-three/rapier:

{characterMeshes.map((mesh, index) => (
            <RigidBody
              ref={rigidRefs.current[index]}
              colliders='hull'
              // restitution={0}
              // isSleeping={true}
              // rotation={[rotX, 0, 0]}
              type="kinematicPosition"
              key={`rigidbody-${index}`}
            >
              <MeshCollider
                  mass={0}
                  sensor
                  onIntersectionEnter={() => { (mesh) => onKeyEnter(mesh) }}
                  onIntersectionExit={() => { (mesh) => onKeyExit(mesh) }}
              >
                <mesh
                  ref={meshRefs.current[index]}
                  key={`${mesh.name}-${index}`}
                  geometry={mesh.geometry}
                  position={[mesh.position.x * scale, mesh.position.y * scale, mesh.position.z * scale]}
                  receiveShadow
                  castShadow
                  material={mesh.material}
                  scale={scale}
                >
                </mesh>
              </MeshCollider>
            </RigidBody>
          ))}