the dispatchGesture doesn't seem to be working

Question

I use the following code to programmatically click on screen. but it just doesn't work. I tested it on Android 13 and Android 14 both debug and release versions

    class ClickAccessibilityService : AccessibilityService() {


    override fun onAccessibilityEvent(event: AccessibilityEvent?) {
        // Log accessibility events
        Log.d("ClickService", "Accessibility Event: ${event?.eventType}")
    }
    
    override fun onInterrupt() {
        // Handle interruptions if needed
    }
    
    override fun onServiceConnected() {
        super.onServiceConnected()
        Log.d("ClickService", "Service Connected")
    }
    
    fun clickAtPosition(x: Int, y: Int) {
        val clickPath = Path()
        clickPath.moveTo(x.toFloat(), y.toFloat())
        //Here we are adding one pixel to the y value so as the give a click some range on where to click.
        clickPath.lineTo(x.toFloat(), y.toFloat() + 1)
        //When providing Stroke Description you should add non zero start time and time for gesture.
        val clickStroke = StrokeDescription(clickPath, 50, 50)

        val clickBuilder = GestureDescription.Builder()
        clickBuilder.addStroke(clickStroke)
        dispatchGesture(clickBuilder.build(), null, null)
        Log.d("ClickService", "Clicked at coordinates: ($x, $y)")
    }

}

The accessibility permissions are granted before execution of the code. and here is the part about it in manifest file:

    <service
        android:name=".customutils.ClickAccessibilityService"
        android:exported="true"
        android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
        <intent-filter>
            <action android:name="android.accessibilityservice.AccessibilityService" />
        </intent-filter>

        <meta-data
            android:name="android.accessibilityservice"
            android:resource="@xml/accessibility_service_config" />
    </service>

the code is invoked as the following:

    val accessibilityService = ClickAccessibilityService()

    accessibilityService.clickAtPosition(result.x.toInt(), result.y.toInt())

    println("click")

accessibility_service_config.xml:

<?xml version="1.0" encoding="utf-8"?>

<accessibility-service
xmlns:android="http://schemas.android.com/apk/res/android"
android:accessibilityEventTypes="typeAllMask"
android:accessibilityFeedbackType="feedbackGeneric"
android:notificationTimeout="100"
android:canRetrieveWindowContent="true"
android:canPerformGestures="true"
android:accessibilityFlags="flagDefault|flagIncludeNotImportantViews"
android:settingsActivity="com.example.ba.SettingsActivity" />

It seems to me that this method doesn't work anymore or something.

Answer 1

When dispatching gestures, you have two essential things that you need to be correct about. First the path you are using for dispatching gestures and the time provided for executing clicks. The essential issue is in the lines below. The code should be modified something like this:

fun clickAtPosition(x: Int, y: Int){
        val clickPath = Path()
        clickPath.moveTo(x.toFloat(), y.toFloat())
        //Here we are adding one pixel to the y value so as the give a click some range on where to click.
        clickPath.lineTo(x.toFloat(), y.toFloat() + 1)
        //When providing Stroke Description you should add non zero start time and time for gesture.
        val clickStroke = StrokeDescription(clickPath, 50, 50)

1. We need to add a lineTo statement to the path because when using moveTo it moves the path's origin to that point but there is no path existing. So we add a line to statement which is a path from the y value to y + 1 value, essentially in a sense a single click.

2. The second problem of time is why 50 milliseconds is used for the start time. This is the duration after which the gesture shall be sent to the screen. This gives ample time for the screen to completely process previous renders and start the click. Gesture time of 50ms is the time for which the single click is done. Using time lower than, say 200ms which we generally use as a threshold for long clicks, is a nice value.

So assuming you have setup your accessibility service correctly, the following changes will make it work.

Edit 1: To check if your gestures are correctly being dispatched, you can use GestureResultCallback. Now my main language is JAVA so you can convert it into Kotlin

                GestureResultCallback callback = new GestureResultCallback() {
                    @Override
                    public void onCompleted(GestureDescription gestureDescription) {
                        Log.d("Gesture","Gesture Completed");
                    }

                    @Override
                    public void onCancelled(GestureDescription gestureDescription) {
                        Log.d("Gesture", "Gesture Cancelled");
                    }
                };
                dispatchGesture(clickBuilder.build(), callback, new Handler(getMainLooper()));

Also, be sure that the coordinates that you are passing has a clickable view underneath. To see the coordinates you can use Developer Tools> Show Pointer Location in Developer Options Android: https://i.sstatic.net/X1KIX.png

With the above tool you can find out the coordinates of the button you need to click.

Edit 2: It seems to me that you are not correctly initializing the Accessibility Service. Accessibility Services are created and bound by the system so you cannot instantiate it as an object in your class. Doing val accessibilityService = ClickAccessibilityService() is wrong. AccessibilityServices are not normal Services. They are started by the system. What you can instead do is in ClickAccessibility class, override onCreate function and then declare a static variable(or Companion in Kotlin) called instance and initialize it to this in the onCreate function. Now you can access this static variable instance and call ClickAccessibility.instance.clickAtPosition() in your class. Be sure to check that instance is null or not. This is also not an elegant solution but for your case it will work. If you want to do it correctly you should use intents to pass data and call functions back and forth between your class and AccessibilityService.