Android Kotlin - How do I determine when "last pointer up" on multitouch for several buttons?

Question

Screenshot

I want to do something like playing chords. Any of these five buttons can be pressed in any combination to provide a different result. The order in which they are pressed does not matter, just the end combination.

Each button click gets registered, but the final total doesn't show until I touch elsewhere on the screen (i.e., not on a button). Any of the one through five buttons can be pressed in any combination, but the result will not be known until all fingers are lifted from the screen (because you never know if there are additional buttons to be pressed for that particular chord.)

I want the chord to play when the last finger is lifted from the buttons, and to zero out the running total after displaying the chord, and not require an extra touch on the screen elsewhere to display the chord or zero it out.

Here is my Main activity:

package com.xxx.multitouchtest

import android.content.Context.MODE_PRIVATE
import android.content.Intent
import android.graphics.Color
import androidx.appcompat.app.AppCompatActivity
import android.os.Bundle
import android.util.Log
import android.view.MotionEvent
import android.view.View
import android.widget.Button
import androidx.constraintlayout.widget.ConstraintLayout


private val TAG: String = "AppName"
var finalLetterValue = 0

var buttonPressed1 = ""
var buttonPressed2 = ""
var buttonPressed3 = ""
var buttonPressed4 = ""
var buttonPressed5 = ""


class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_motion_screen)
        val viewScr = findViewById(R.id.activity_motion_screen) as ConstraintLayout
        val textLine = findViewById<View>(R.id.editTextTextMultiLine)
        val buttonReset = findViewById<View>(R.id.buttonCalibrate)
        val buttonShowKey = findViewById<View>(R.id.buttonShowKey)
        val buttonThumb = Button(this)
        val buttonIndex = Button(this)
        val buttonMiddle = Button(this)
        val buttonRing = Button(this)
        val buttonLittle = Button(this)

        //defines Preferences file and access mode -- private = 0
        val sharedPreferences = getSharedPreferences("prefFile", MODE_PRIVATE)
        //initializes editor
        val editor = sharedPreferences.edit()

        //extract saved key
        var isItCalibrated = sharedPreferences.getString("calibratedState", "")
        if (isItCalibrated.isNullOrEmpty()) {
            Log.i(TAG, "its empty h $isItCalibrated")
            editor.putString("calibratedState", "N")
            editor.apply()
            isItCalibrated = "N" // init arbitrarily to NO if needed

            viewScr.removeAllViews();
            viewScr.addView(textLine);
            viewScr.addView(buttonReset);
            viewScr.addView(buttonShowKey);

            //send to Calibrate screen first - do this later
            //val intent = Intent(this, Calibrate::class.java)
            //startActivity(intent)

        } else if (isItCalibrated == "N") {
            viewScr.removeAllViews();
            viewScr.addView(textLine);
            viewScr.addView(buttonReset);
            viewScr.addView(buttonShowKey);
        } else if (isItCalibrated == "Y") {
            //place buttons into saved positions on screen
            var thumbPosx = sharedPreferences.getFloat("thumbPosx", 0F)
            var thumbPosy = sharedPreferences.getFloat("thumbPosy", 0F)
            var indexPosx = sharedPreferences.getFloat("indexPosx", 0F)
            var indexPosy = sharedPreferences.getFloat("indexPosy", 0F)
            var middlePosx = sharedPreferences.getFloat("middlePosx", 0F)
            var middlePosy = sharedPreferences.getFloat("middlePosy", 0F)
            var ringPosx = sharedPreferences.getFloat("ringPosx", 0F)
            var ringPosy = sharedPreferences.getFloat("ringPosy", 0F)
            var littlePosx = sharedPreferences.getFloat("littlePosx", 0F)
            var littlePosy = sharedPreferences.getFloat("littlePosy", 0F)
            Log.i(TAG, "its calibrated $isItCalibrated")

            buttonThumb.x = thumbPosx
            buttonThumb.y = thumbPosy
            buttonThumb.setBackgroundColor(Color.YELLOW)
            buttonThumb.setTextColor(Color.BLACK)
            viewScr.addView(buttonThumb);
            //buttonThumb.text = "$id1 thumb" //--this is shown at calibration time

            buttonIndex.x = indexPosx
            buttonIndex.y = indexPosy
            buttonIndex.setBackgroundColor(Color.BLUE)
            buttonIndex.setTextColor(Color.YELLOW)
            viewScr.addView(buttonIndex);

            buttonMiddle.x = middlePosx
            buttonMiddle.y = middlePosy
            buttonMiddle.setBackgroundColor(Color.BLUE)
            buttonMiddle.setTextColor(Color.YELLOW)
            viewScr.addView(buttonMiddle);

            buttonRing.x = ringPosx
            buttonRing.y = ringPosy
            buttonRing.setBackgroundColor(Color.BLUE)
            buttonRing.setTextColor(Color.YELLOW)
            viewScr.addView(buttonRing);

            buttonLittle.x = littlePosx
            buttonLittle.y = littlePosy
            buttonLittle.setBackgroundColor(Color.BLUE)
            buttonLittle.setTextColor(Color.YELLOW)
            viewScr.addView(buttonLittle);
        } else {
            Log.i(TAG, "not empty $isItCalibrated")
        }




        buttonReset.setOnClickListener(View.OnClickListener {
            viewScr.removeAllViews();
            viewScr.addView(textLine);
            viewScr.addView(buttonReset);
            viewScr.addView(buttonShowKey);
            editor.putString("calibratedState", "N")
            editor.apply()
            isItCalibrated = "N"
        })

        //send to ShowKey screen
        buttonShowKey.setOnClickListener(View.OnClickListener {
            val intent = Intent(this, ShowKey::class.java)
            startActivity(intent)
        })

//==========================================================

        viewScr.setOnTouchListener{
            v: View,
            m: MotionEvent -> handleTouch(m)
            true
        }
    }



    private fun handleTouch(m: MotionEvent)
    {
        var pointerCount = m.pointerCount
        Log.d(" ", "actual pointerCount $pointerCount")

        //==============
        //seems to be an issue with pointerCount - is it not totalling?
        //where does it go up (and down?)
        //??do I need to COUNT the pointerCount -- running total?????
        // looks like it is ALWAYS 1!
        //??is m.getPointerId(i) the value I need?
        //--  It is incrementing these counters only outside of the buttonclicked
        //      sections.

        for (i in 0 until pointerCount) {

            Log.d(" ", "for pointerCount $pointerCount")

            val x = m.getX(i)
            val y = m.getY(i)
            var id = m.getPointerId(i) //range 0-4
            var id1 = id + 1 //changing range 0-4 to 1-5
            val action = m.actionMasked
            val actionIndex = m.actionIndex
            var actionString: String

            /*
            ACTION_DOWN is for the first finger that touches the screen.
            This starts the gesture. The pointer data for this finger is always at index 0 in
            the MotionEvent.
            ACTION_POINTER_DOWN is for extra fingers that enter the screen beyond the first.
            The pointer data for this finger is at the index returned by getActionIndex().
            ACTION_POINTER_UP is sent when a finger leaves the screen but at least one finger is still touching it.
            The last data sample about the finger that went up is at the index returned by getActionIndex().
            ACTION_UP is sent when the last finger leaves the screen.
            The last data sample about the finger that went up is at index 0. This ends the gesture.
            ACTION_CANCEL means the entire gesture was aborted for some reason. This ends the gesture.
            */

            when (action) {
                MotionEvent.ACTION_DOWN -> actionString = "first pointer DOWN"
                MotionEvent.ACTION_UP -> actionString = "last pointer UP"
                MotionEvent.ACTION_POINTER_DOWN -> actionString = "next PNTR DOWN"
                MotionEvent.ACTION_POINTER_UP -> actionString = "next PNTR UP"
                MotionEvent.ACTION_MOVE -> actionString = "MOVE"
                else -> actionString = "none"
            }
//


            val touchStatus =
                "i: $i Action: $actionString Index: $actionIndex  Pointer: $id  X: $x  Y: $y"
            Log.d(" ", touchStatus)
            Log.d("FINAL TOTAL (FLV) ", finalLetterValue.toString())
            Log.d("pointerCount ", pointerCount.toString())
            // check buttons pressed
            if (actionString == "last pointer UP") {
                if (buttonPressed1 == "y") {
                    Log.d(" ", "1y")
                }
                if (buttonPressed2 == "y") {
                    Log.d(" ", "2y")
                }
                if (buttonPressed3 == "y") {
                    Log.d(" ", "3y")
                }
                if (buttonPressed4 == "y") {
                    Log.d(" ", "4y")
                }
                if (buttonPressed5 == "y") {
                    Log.d(" ", "5y")
                }
                buttonPressed1 = "n"
                buttonPressed2 = "n"
                buttonPressed3 = "n"
                buttonPressed4 = "n"
                buttonPressed5 = "n"
                Log.d(" ", "button presses reset to n")

            }


            //defines Preferences file and access mode -- private = 0
            val sharedPreferences = getSharedPreferences("prefFile", MODE_PRIVATE)
            //initializes editor
            val editor = sharedPreferences.edit()
            //for calibration section check
            var isItCalibrated = sharedPreferences.getString("calibratedState", "")

            var thumbPosx = sharedPreferences.getFloat("thumbPosx", 0F)
            var thumbPosy = sharedPreferences.getFloat("thumbPosy", 0F)
            var indexPosx = sharedPreferences.getFloat("indexPosx", 0F)
            var indexPosy = sharedPreferences.getFloat("indexPosy", 0F)
            var middlePosx = sharedPreferences.getFloat("middlePosx", 0F)
            var middlePosy = sharedPreferences.getFloat("middlePosy", 0F)
            var ringPosx = sharedPreferences.getFloat("ringPosx", 0F)
            var ringPosy = sharedPreferences.getFloat("ringPosy", 0F)
            var littlePosx = sharedPreferences.getFloat("littlePosx", 0F)
            var littlePosy = sharedPreferences.getFloat("littlePosy", 0F)

            //Pointer id in order of touch 1-5
            Log.d(" ", " ")
            Log.d(" ", "touch id $id1")
            Log.d(" ", "$actionString")
            val coLayout = findViewById(R.id.activity_motion_screen) as ConstraintLayout
            val textLine = findViewById<View>(R.id.editTextTextMultiLine)
            val buttonReset = findViewById<View>(R.id.buttonCalibrate)
            val buttonShowKey = findViewById<View>(R.id.buttonShowKey)
            val buttonThumb = Button(this)
            val buttonIndex = Button(this)
            val buttonMiddle = Button(this)
            val buttonRing = Button(this)
            val buttonLittle = Button(this)
/*
            var buttonPressed1 = "n"
            var buttonPressed2 = "n"
            var buttonPressed3 = "n"
            var buttonPressed4 = "n"
            var buttonPressed5 = "n"
*/
            val thumbValue  = 1
            val indexValue  = 2
            val middleValue = 3
            val ringValue   = 4
            val littleValue = 5

            // =======================================
            //calculate all touched button Values

            //want to have these running all the time, and
            //not reset until last pointer is up, but how to see that?

            buttonThumb.setOnClickListener {
                //testClick()
                buttonPressed1 = "y"
                //pointerCount = pointerCount + 1

                finalLetterValue = finalLetterValue + thumbValue
                Log.d(" ", " ")
                Log.d("actionString in thumb ", "$actionString")
                Log.d("thumb CLICKED ", " ")
                Log.d("thumb VALUE ", thumbValue.toString())
                Log.d("running TOTAL (FLV) ", finalLetterValue.toString())
            }
            buttonIndex.setOnClickListener {
                buttonPressed2 = "y"
                //pointerCount = pointerCount + 1

                finalLetterValue = finalLetterValue + indexValue
                Log.d(" ", " ")
                Log.d("actionString in index ", "$actionString")
                Log.d("index CLICKED ", " ")
                Log.d("index VALUE ", indexValue.toString())
                Log.d("running TOTAL (FLV) ", finalLetterValue.toString())
                if (actionString =="last pointer UP")
                {
                    Log.d("final letterValue in index up", finalLetterValue.toString())
                    //finalLetterValue = 0
                }
            }
            buttonMiddle.setOnClickListener {
                buttonPressed3 = "y"
                finalLetterValue = finalLetterValue + middleValue
                Log.d(" ", " ")
                Log.d("actionString in middle ", "$actionString")
                Log.d("middle CLICKED ", " ")
                Log.d("middle VALUE ", middleValue.toString())
                Log.d("running TOTAL (FLV) ", finalLetterValue.toString())
            }
            buttonRing.setOnClickListener {
                buttonPressed4 = "y"
                finalLetterValue = finalLetterValue + ringValue
                Log.d(" ", " ")
                Log.d("actionString in ring ", "$actionString")
                Log.d("ring CLICKED ", " ")
                Log.d("ring VALUE ", ringValue.toString())
                Log.d("running TOTAL (FLV) ", finalLetterValue.toString())
            }
            buttonLittle.setOnClickListener {
                buttonPressed5 = "y"
                finalLetterValue = finalLetterValue + littleValue
                Log.d(" ", " ")
                Log.d("actionString in little ", "$actionString")
                Log.d("little CLICKED ", " ")
                Log.d("little VALUE ", littleValue.toString())
                Log.d("running TOTAL (FLV) ", finalLetterValue.toString())
                }


            Log.d("LAST action outside of clicks  ", actionString)

            if (actionString =="last pointer UP")
            {
                Log.d("final letterValue in LPU", finalLetterValue.toString())
                finalLetterValue = 0
            }

            if (actionString == "first pointer DOWN")
                {
                    Log.d("final letterValue in FPD", finalLetterValue.toString())
                    //finalLetterValue = 0
                }


            } //end-actionString check
            } //end-for
        } //end-HandleMotion
    }

Here's what happens when I press 4 and 5 (ring and little fingers), then lift all.

D/actionString in ring: first pointer DOWN
D/ring CLICKED:  
D/ring VALUE: 4
D/running TOTAL (FLV): 4
I/ViewRootImpl@a632ff6[MainActivity]: ViewPostIme pointer 1
I/MSHandlerLifeCycle: isMultiSplitHandlerRequested: windowingMode=1 isFullscreen=true isPopOver=false isHidden=false skipActivityType=false isHandlerType=true this: DecorView@2f39ce4[MainActivity]
D/:  
D/actionString in little: first pointer DOWN
D/little CLICKED:  
D/little VALUE: 5
D/running TOTAL (FLV): 9

The running total is 9, which is correct. It should then play chord "9" and reset. This is the major issue. It doesn't register "last pointer UP" here.

However, touching another button (the "3") adds that one to the total. See here - first pointer DOWN, but running total is 12 when it should be "3" only, on a new chord.

D/:  
D/actionString in middle: first pointer DOWN
D/middle CLICKED:  
D/middle VALUE: 3
D/running TOTAL (FLV): 12

It's not until I touch somewhere else on the screen that I get the final total and reset I want. (I didn't touch any of the buttons, just a blank spot on the screen.)

I/ViewRootImpl@a632ff6[MainActivity]: ViewPostIme pointer 0
I/MSHandlerLifeCycle: isMultiSplitHandlerRequested: windowingMode=1 isFullscreen=true isPopOver=false isHidden=false skipActivityType=false isHandlerType=true this: DecorView@2f39ce4[MainActivity]
D/: actual pointerCount 1
D/: for pointerCount 1
D/: i: 0 Action: first pointer DOWN Index: 0  Pointer: 0  X: 380.75  Y: 361.74023
D/FINAL TOTAL (FLV): 12
D/pointerCount: 1
D/:  
D/: touch id 1
D/: first pointer DOWN
D/LAST action outside of clicks: first pointer DOWN
D/final letterValue in FPD: 12
I/MSHandlerLifeCycle: isMultiSplitHandlerRequested: windowingMode=1 isFullscreen=true isPopOver=false isHidden=false skipActivityType=false isHandlerType=true this: DecorView@2f39ce4[MainActivity]
D/: actual pointerCount 1
D/: for pointerCount 1
D/: i: 0 Action: MOVE Index: 0  Pointer: 0  X: 376.64844  Y: 364.11328
D/FINAL TOTAL (FLV): 12
D/pointerCount: 1
D/:  
D/: touch id 1
D/: MOVE
D/LAST action outside of clicks: MOVE
I/ViewRootImpl@a632ff6[MainActivity]: ViewPostIme pointer 1
I/MSHandlerLifeCycle: isMultiSplitHandlerRequested: windowingMode=1 isFullscreen=true isPopOver=false isHidden=false skipActivityType=false isHandlerType=true this: DecorView@2f39ce4[MainActivity]
D/: actual pointerCount 1
D/: for pointerCount 1
D/: i: 0 Action: last pointer UP Index: 0  Pointer: 0  X: 377.23438  Y: 364.11328
D/FINAL TOTAL (FLV): 12
D/pointerCount: 1
D/: 3y
D/: 4y
D/: 5y
D/: button presses reset to n
D/:  
D/: touch id 1
D/: last pointer UP
D/LAST action outside of clicks: last pointer UP
D/final letterValue in LPU: 12

Answer 1

You're using multiple touch handlers here - the buttons themselves, through the OnClickListeners you're setting on them, and the OnTouchListener you're setting on the background layout.

These are completely separate gestures - see how when you put the first finger down, your logs only show output from the click listener, not the touch listener on the layout? So the touch listener has no idea a gesture has started - the button is consuming that touch event, and not passing it along for another View to handle.

The other thing is, when you put the second finger down, it's being treated as a separate gesture, because you're tapping another button and that event is being handled independently of the first button. So the second button has no idea that the first is already pressed, and the OnTouchListener has no idea any touches are happening at all.

Then when you tap somewhere else, then the OnTouchListener kicks in, gets the ACTION_UP when you finish the tap, and calculates the thing (which you can probably break by tapping a button multiple times so it keeps adding its value)

---

There are a few different approaches to this - you could use Buttons and have them talk to a separate component that keeps track of which ones said they're pressed, when the last button has said "I'm not pressed anymore" so it can fire off the "play these notes" event, that kind of thing.

Alternatively, you could drop the buttons (and their independent, click-stealing behaviour) entirely, and create your own custom view that draws "buttons" on its Canvas and handles all the touches on its surface. And it seems like that's what you want to do, since you're already writing an OnTouchListener that's meant to keep track of gestures and which pointers are down, etc!

If you still want to do things this way, you'd have to make the Buttons forward their touch events to the parent layout. If you subclass Button, that could be a little complicated. You could try experimenting with making your OnTouchListener an object and setting that same instance on your layout and each button (as a touch listener, not a click listener), make onTouch return false if the current view isn't one of the buttons (so they don't consume the event), and see if they all all handle the same event/gesture. Or maybe the buttons can have separate OnTouchListeners so long as they pass false - I'm not sure how all this would work stuck on top of a Button, so I'm just giving you things to try

---

Personally I'd go with the "buttons talk to a central component that works out what's happening" approach unless you're up for doing the full custom view thing (which would give you more control and flexibility). You'd still need a way to know when a button is "pressed" and when it's "released" though, so you'll probably need OnTouchListeners on them anyway to fire off those events