CODEWITHSUNDEEP
androidandroid-viewaccessibility
Fresh Prince
Fresh Prince

Reputation: 189

How to extract information about UI elements in an android app at runtime?

Hierarchy Viewer lets you extract information about the UI elements by connecting your phone to the laptop. It works by polling the Window Manager/View Manager for information via adb.

I was wondering if it is possible to extract UI information of the foreground screen on the phone at runtime using a background service?

I am trying to build an accessibility service (similar to Talkback), and was trying to find a way to collect information about all the UI elements (their class (e.g. button) and location) on the active foreground screen. I have been able to make an accessibility service, but I have not been able to find a way to get the UI info.

Upvotes: 2

Views: 2755

Answers (2)

Fresh Prince
Fresh Prince

Reputation: 189

I was able to find an answer to my problem.

The Android accessibility API provides this functionality. I was able to implement an Accessibility Service and for each accessibility event, I was able to use the Accessibility Node Info functions, which is essentially a tree representation of all the UI elements on the foreground screen. For example A linear layout consists of a button and an image, then the button and the image would be children for the LL.

The root of the tree can be found by: 

AccessibilityNodeInfo rootNode = getRootInActiveWindow();

Then I was able to traverse the tree (Breadth first search) using the getChild() function which gives me a list of all the UI elements on the active screen.

Hope it helps someone else too!

Upvotes: 2

Ruize Zhang
Ruize Zhang

Reputation: 76

I wrote some code and works fine, maybe it's what you want.

Firstly, get the content view:

ViewGroup parent = (ViewGroup) findViewById(android.R.id.content);

Then use a method to check every element of the parent view:

private void showViewInfo(View view){
    if(view instanceof ViewGroup){

        //do something when find view group:
        System.out.println(view.getClass().toString()+"\tx:"+view.getX()+"\ty:"+view.getY());


        for(int i=0;i<((ViewGroup) view).getChildCount();i++){
            showViewInfo(((ViewGroup) view).getChildAt(i));
        }
    }else{

        //do something when find children
        System.out.println(view.getClass().toString()+"\tx:"+view.getX()+"\ty:"+view.getY());
    }
}

Finally, call the method with the parent view as parameter:

showViewInfo(parent);

and you can get something like this:

System.out: class android.support.v7.widget.ContentFrameLayout  x:0.0   y:0.0

Here is whole code:

public class MainActivity extends AppCompatActivity {

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);


    ViewGroup parent = (ViewGroup) findViewById(android.R.id.content);
    showViewInfo(parent);
}

private void showViewInfo(View view){
    if(view instanceof ViewGroup){
        //do something when find view group:
        System.out.println(view.getClass().toString()+"\tx:"+view.getX()+"\ty:"+view.getY());
        //find the children
        for(int i=0;i<((ViewGroup) view).getChildCount();i++){
            showViewInfo(((ViewGroup) view).getChildAt(i));
        }
    }else{
        //do something when find children
        System.out.println(view.getClass().toString()+"\tx:"+view.getX()+"\ty:"+view.getY());
    }
}

}

Upvotes: 1

Related Questions