Custom snap gesture in visionOS

Question

I’ve been developing a snap gesture for my app. While it works, it’s inconsistent and activates mistakenly.

Per Wikipedia, a snap occurs when tension is created with the index, middle, or ring finger against the thumb and releases with force in 7 milliseconds, producing sound.

To replicate this, I tried tracking contact between the finger and thumb, then check if the finger is close to points 1 and 5 in the diagram below upon release.

The current implementation "kind of works" but lacks consistency for seamless user interaction. For example, if the .thumbTip and .indexFingerTip are touching, and the hand unintentionally closes, the gesture activates incorrectly, which isn’t ideal. In which way can I create something robust and consistent?

Answer 1

I figured a 90% effective code, might not be the most efficient but it works for my needs. If anyone feels like improving it, feel free:

private func snapGestureActivated(for handSide: String, finger: HandSkeleton.JointName) -> Bool {
    guard let handAnchor = (handSide == "left" ? latestHandTracking.left : latestHandTracking.right),
          let handSkeleton = handAnchor.handSkeleton,
          handAnchor.isTracked else {
        resetState()
        return false
    }
    
    let origin = handAnchor.originFromAnchorTransform
    let joint = handSkeleton.joint
    
    let thumbAnchor = joint(.thumbTip).anchorFromJointTransform
    let fingerAnchor = joint(finger).anchorFromJointTransform
    let thumbKnuckleAnchor = joint(.thumbKnuckle).anchorFromJointTransform
    
    let thumbPosition = matrix_multiply(origin, thumbAnchor).columns.3.xyz
    let fingerPosition = matrix_multiply(origin, fingerAnchor).columns.3.xyz
    let thumbKnucklePosition = matrix_multiply(origin, thumbKnuckleAnchor).columns.3.xyz
    
    let distanceThumbFinger = simd_precise_distance(thumbPosition, fingerPosition)
    let distanceFingerDestination = simd_precise_distance(fingerPosition, thumbKnucklePosition)
    
    let contactThreshold: Float = 0.011
    let destinationThreshold: Float = 0.08
    
    let currentTime = Date().timeIntervalSince1970
    
    if distanceThumbFinger < contactThreshold {
        detectedContactTime = currentTime
        
        switch finger {
        case .indexFingerTip:
            updateContacts(for: handSide, index: true, middle: false, ring: false)
        case .middleFingerTip:
            updateContacts(for: handSide, index: false, middle: true, ring: false)
        case .ringFingerTip:
            updateContacts(for: handSide, index: false, middle: false, ring: true)
        default:
            break
        }
        
        if shouldLog() {
            print("Phase 1: \(handSide) \(finger) touched.")
        }
    }
    if isContactFlagActive(for: handSide, finger: finger) &&
        distanceFingerDestination < destinationThreshold &&
        (currentTime - detectedContactTime) < 0.15 {
        resetState()
        print("Phase 2: Snap gesture detected with \(finger).")
        return true
    }
    
    return false
}

These are the helper functions I used:

private func updateContacts(for handSide: String, index: Bool, middle: Bool, ring: Bool) {
    if handSide == "left" {
        leftContactIndex = index
        leftContactMiddle = middle
        leftContactRing = ring
        rightContactIndex = false
        rightContactMiddle = false
        rightContactRing = false
    } else {
        rightContactIndex = index
        rightContactMiddle = middle
        rightContactRing = ring
        leftContactIndex = false
        leftContactMiddle = false
        leftContactRing = false
    }
}

private func isContactFlagActive(for handSide: String, finger: HandSkeleton.JointName) -> Bool {
    switch finger {
    case .indexFingerTip:
        return handSide == "left" ? leftContactIndex : rightContactIndex
    case .middleFingerTip:
        return handSide == "left" ? leftContactMiddle : rightContactMiddle
    case .ringFingerTip:
        return handSide == "left" ? leftContactRing : rightContactRing
    default:
        return false
    }
}