Swept collision of rotated rectangle and moving collider

Question

In the game I'm developing I have a character collider, which can collider with various objects in the world. The character collider has a center position, a size in the x and z direction and a height in the y direction. Because I don't care about it's exact collider shape it is used as if it'd be an axis aligned bounding box, however I do not care about the orientation of the axes. The obstacles in the world also have bounding boxes, however they also have a rotation around the y axis. I wanted to implement swept AABB collision detection, so that for each potential collider (checked in a broadphase before this) the players' collider is transformed into the local space of the collider and the swept collision detection is performed in that space.

Now my problem: My implementation does not work, and I spent days already pondeing over it without finding a solution. It works fine when the obstacles are not rotated and my character collides and sldes along them like i want, but with rotation the character just passes through them, only occasionally it collides, but also just for a bit and then goes through.

Here is my implementation in one function:

fun getMovementAfterCollision(collider: MovableCollider, obstacle: Collider3D, velocityX: Float, velocityY: Float, velocityZ: Float, callback: (Float, Float, Float) -> Unit) {
    var colliderMinX = collider.minX
    val colliderMinY = collider.minY
    var colliderMinZ = collider.minZ

    var colliderMaxX = collider.maxX
    val colliderMaxY = collider.maxY
    var colliderMaxZ = collider.maxZ

    var obstacleMinX = obstacle.minX
    val obstacleMinY = obstacle.minY
    var obstacleMinZ = obstacle.minZ

    var obstacleMaxX = obstacle.maxX
    val obstacleMaxY = obstacle.maxY
    var obstacleMaxZ = obstacle.maxZ

    var moveX = velocityX
    var moveY = velocityY
    var moveZ = velocityZ

    if (obstacle.rotation != 0.0f) {
        val cos = cos(toRadians(obstacle.rotation))
        val sin = sin(toRadians(obstacle.rotation))

        val localColliderX = collider.x - obstacle.centerX
        val localColliderZ = collider.z - obstacle.centerZ

        val rotatedColliderX = cos * localColliderX - sin * localColliderZ
        val rotatedColliderZ = sin * localColliderX + cos * localColliderZ

        colliderMinX = rotatedColliderX - collider.size * 0.5f
        colliderMinZ = rotatedColliderZ - collider.size * 0.5f
        colliderMaxX = rotatedColliderX + collider.size * 0.5f
        colliderMaxZ = rotatedColliderZ + collider.size * 0.5f

        obstacleMinX = -obstacle.width * 0.5f
        obstacleMinZ = -obstacle.depth * 0.5f
        obstacleMaxX = obstacle.width * 0.5f
        obstacleMaxZ = obstacle.depth * 0.5f

        moveX = cos * velocityX - sin * velocityZ
        moveZ = sin * velocityX + cos * velocityZ
    }

    val epsilon = 1e-5f

    if (moveX > epsilon)
        moveX = min(moveX, obstacleMinX - colliderMaxX)
    else if (moveX < -epsilon)
        moveX = max(moveX, obstacleMaxX - colliderMinX)

    if (moveY > epsilon)
        moveY = min(moveY, obstacleMinY - colliderMaxY)
    else if (moveY < -epsilon)
        moveY = max(moveY, obstacleMaxY - colliderMinY)

    if (moveZ > epsilon)
        moveZ = min(moveZ, obstacleMinZ - colliderMaxZ)
    else if (moveZ < -epsilon)
        moveZ = max(moveZ, obstacleMaxZ - colliderMinZ)

    if (obstacle.rotation != 0.0f) {
        val cos = cos(toRadians(-obstacle.rotation))
        val sin = sin(toRadians(-obstacle.rotation))

        val rotatedMoveX = cos * moveX - sin * moveZ
        val rotatedMoveZ = sin * moveX + cos * moveZ

        moveX = rotatedMoveX
        moveZ = rotatedMoveZ
    }

    val allowedDX = min(abs(moveX), abs(velocityX)) * sign(velocityX)
    val allowedDY = min(abs(moveY), abs(velocityY)) * sign(velocityY)
    val allowedDZ = min(abs(moveZ), abs(velocityZ)) * sign(velocityZ)

    callback(allowedDX, allowedDY, allowedDZ)
}

I was thinking of using SAT instead, but in my eyes it should be possible to use this method given my restrictions (the 3d collision can be basically viewed as a 2d problem because of the rotation only in the y axis). I just can't what's wrong with my implementation.