How to handle MotionEvents (Gamepad Joystick Movement, etc) from AccessibilityService

Question

I have an AccessibilityService that takes in input from game controllers (ps5 controller, xbox controller, etc.).

I use the onKeyEvent() method to handle button presses and releases, so I can handle those easily. The problem I am facing is how to handle the Joystick movements and top trigger presses, as I am unaware how to handle them through an AccessibilityService.

Normally, I would simply use onGenericMotionEvent() to handle these MotionEvents, but unfortunately AccessibilityService does not provide me with such a method. I have looked at the docs and official codelabs for almost 3 weeks with no luck, if someone could tell me how to handle MotionEvents through an AccessibilityService I would be very relieved.

The MotionEvents I want to handle are these:

AXIS_X, AXIS_Y, AXIS_Y, AXIS_RZ, AXIS_RY, AXIS_RX, AXIS_HAT_X, AXIS_HAT_Y, AXIS_LTRIGGER, AXIS_RTRIGGER, AXIS_BRAKE,AXIS_GAS.

There may be others depending on the controller, but these are the main ones I need to handle input from my controller.

Regards, 0xB01b

Answer 1

To create Accessibility Service that can combine with Input Method Service, you can try the following snippet:

var tapService: TapService? = null

class TapService : AccessibilityService() {
    companion object {
        private const val TAG = "TapService"
    }

    override fun onAccessibilityEvent(event: AccessibilityEvent?) { }

    override fun onInterrupt() { }

    override fun onServiceConnected() {
        super.onServiceConnected()
        Log.d(TAG, "onServiceConnected")

        tapService = this

        startActivity(Intent(this, MainActivity::class.java).addFlags(Intent.FLAG_ACTIVITY_NEW_TASK))
    }

    override fun onUnbind(intent: Intent?): Boolean {
        Log.d(TAG, "onUnbind")
        tapService = null
        return super.onUnbind(intent)
    }

    override fun onDestroy() {
        Log.d(TAG, "onDestroy")
        tapService = null
        super.onDestroy()
    }

    fun tap(x: Int, y: Int, hold: Boolean) {
        // create path and build the GestureDescription then dispatch it
    }

    fun swipe(fromX: Int, fromY: Int, toX: Int, toY: Int, duration: Long) {
        // similar to tap function
    }
}

when the user starts the TapService in the Accessibility Setting Menu, it calls the function onServiceConnected and assigns the tapService object to the service.

Hence, you can call the method tapService.tap() or tapService.swipe() elsewhere in the Input Method Service to inject the touch action.

However, because of the limitation from Accessibility Service, the simulation of touch action is not as expected.

Answer 2

To create an InputMethodService, you can refer to this official document.

First, add service in the Manifest.xml:

<service android:name=".IMService"
        android:permission="android.permission.BIND_INPUT_METHOD">
        <intent-filter>
            <action android:name="android.view.InputMethod"/>
        </intent-filter>
        <meta-data
            android:name="android.view.im"
            android:resource="@xml/method"/>
    </service>

Next, method.xml is need to create under res/xml/ folder:

    <?xml version="1.0" encoding="utf-8"?>
<input-method xmlns:android="http://schemas.android.com/apk/res/android"
    android:isDefault="false"
    android:settingsActivity=".MainActivity">
    <subtype
        android:icon="@mipmap/ic_launcher"
        android:imeSubtypeLocale="en_US"
        android:imeSubtypeMode="keyboard"
        android:label="@string/app_name"/>
</input-method>

Layout of the keyboard is not needed in this case.

Then, create the class IMService implementing the InputMethodService.

The following code is written in Kotlin (for example):

class IMService: InputMethodService() {
companion object {
    private const val TAG = "IMService"
}

override fun onKeyDown(keyCode: Int, event: KeyEvent): Boolean {
    if(event.action == KeyEvent.ACTION_DOWN) {
        if (event.source and InputDevice.SOURCE_GAMEPAD == InputDevice.SOURCE_GAMEPAD) {
            // process key down event here
            return true   // return true if you want the key event to be filtered
        }
    }
    return super.onKeyDown(keyCode, event)
}

override fun onKeyUp(keyCode: Int, event: KeyEvent): Boolean {
    if(event.action == KeyEvent.ACTION_UP) {
        if (event.source and InputDevice.SOURCE_GAMEPAD == InputDevice.SOURCE_GAMEPAD) {
            // process key up event here
            return true
        }
    }
    return super.onKeyUp(keyCode, event)
}

override fun onGenericMotionEvent(event: MotionEvent): Boolean {
    if(event.action == MotionEvent.ACTION_MOVE) {
        if (event.source and InputDevice.SOURCE_JOYSTICK == InputDevice.SOURCE_JOYSTICK) {
            // process motion event here
            return true
        }
    }
    return super.onGenericMotionEvent(event)
}
}

The service lifecycle starts after you enabled and changed the default input method in Settings/../Language and keyboard/Keyboard list and default.

Answer 3

In Accessibility Service, you can only get and filter KeyEvents, the MotionEvents can not be handled inside that Service

One possible way is create a Service implement InputMethodService interface. It similar to create a virtual keyboard. You can handle KeyEvent or MotionEvent in the Service globally. What I mean is you can know whatever the input is in every activity after you enable the InputMethodService (like changing the keyboard input). And then, the Accessibility Service can be used to dispatch a Touch or Swipe.

That combination (InputMethodService + AccessibilityService) enables you to map a joystick to Touch and Swipe globally.

However, to make it act like actual joystick mapper, I think the Accessibility Service can not help you to simulate the MotionEvent as you expected. Maybe only the InjectInputEvent can do exactly what you want. But it comes to an old problem: INJECT_EVENTS permission.

Answer 4

The API doesn't support these because AccessibilityServices can only filter KeyEvents, and the buttons as you say do not produce KeyEvents.

Can you explain what you're building? Understanding the impact of adding this api to the lives of people with disabilities would help us prioritize this work along with other stuff we're planning.