Simultaneous drag & long press cancels animation

Question

I'm trying to create a view that can be both dragged and long pressed. At a high level, a basic Circle that:

1. When pressed for N seconds, becomes a button that can perform some action.
2. As the press gesture is progressing, animate the circle to grow in size. Resets if released before N seconds.
3. Draggable throughout, unless it became a button and is released. i.e. you can still drag it if you started the drag before it's a button.

Here's a rough sketch of the state transitions:

I'm almost able to achieve this with the following MRE:

struct DragGestureView: View {
    @State private var buttonSize = 100.0
    @State private var position = CGPoint(x: 100, y: 100)
    let N = 3.0
    
    var body: some View {
        Circle()
            .stroke(.white, lineWidth: 5)
            .overlay(Circle().fill(.blue))
            .frame(width: buttonSize, height: buttonSize)
            .position(position)
            .onLongPressGesture(minimumDuration: N, maximumDistance: .infinity) {
                print("Turn into a button")
            } onPressingChanged: { inProgress in
                if inProgress {
                    withAnimation(.smooth(duration: N)) {
                        buttonSize = 200
                    }
                } else {
                    withAnimation(.bouncy) {
                        buttonSize = 100
                    }
                }
            }
            .simultaneousGesture(
                DragGesture()
                    .onChanged { v in
                        position = v.location
                    }
            )
    }
}

I'm using the onPressingChanged method to animate the circle growing in size and reset if released before N seconds. And I'm setting maximumDistance to .infinity to allow dragging while the long press is in progress.

However, dragging the circle at anytime immediately completes the in progress animation and sets the circle's size to 200 and I can't seem to find a way to prevent this. Anyone have any ideas?