Invoking submenu item using UIAutomationClient cannot find submenu items

Question

I have seen this question asked and answered with different frameworks, but not using the C# and the Interop.UIAutomationClient and .NET 8.0. The answer is always the same, make sure to Expand the menu item before finding submenu items. I am doing that, since this is code that is being moved from a .NET Windows application using AutomationElement into a console app where my options seem to be using COM object for automation. I have written a couple of classes to do the work.

A window automation element wrapper classes that supports some patterns on an element.

    public class WindowElement
    {
        private IUIAutomationElement? element;

        public WindowElement(IUIAutomationElement element) 
        { 
            this.element = element;
        }

        public WindowElement()
        {
        }

        public bool IsValid()
        {
            return element != null;
        }

        public IUIAutomationElement GetElement() 
        {
            return element;
        }

        public bool Expand()
        {
            if (element != null)
            {
                IUIAutomationExpandCollapsePattern pattern = element.GetCurrentPattern(WindowAutomation.UIA_ExpandCollapsePatternId);
                pattern.Expand();
                return pattern.CurrentExpandCollapseState == ExpandCollapseState.ExpandCollapseState_Expanded;
            }
            return false;
        }

        public bool Invoke()
        {
            if (element != null)
            {
                IUIAutomationInvokePattern pattern = element.GetCurrentPattern(WindowAutomation.UIA_InvokePatternId);
                pattern.Invoke();
                return true;
            }
            return false;
        }

        public bool Select()
        {
            if (element != null)
            {
                IUIAutomationSelectionItemPattern pattern = element.GetCurrentPattern(WindowAutomation.UIA_SelectionItemPatternId);
                pattern.Select();
                return true;
            }
            return false;
        }
    }

A class made to find elements from by finding a known application by process id (for sub-process launch) or by a known program name.

public class WindowAutomation
{
    private CUIAutomation8 uia = new CUIAutomation8();

    public WindowElement FindProcessNameElement(string processName)
    {
        IUIAutomationElement root = uia.GetRootElement();
        IUIAutomationElementArray children = root.FindAll(TreeScope.TreeScope_Children, uia.CreateTrueCondition());
        for (int i = 0; i < children.Length; i++)
        {
            IUIAutomationElement child = children.GetElement(i);
            string childName = child.CurrentName;
            if (childName.StartsWith(processName))
            {
                return new WindowElement(child);
            }
        }
        return new WindowElement();
    }

    public WindowElement FindMenuBarItem(WindowElement appElement, string itemName) 
    {
        if (appElement.IsValid())
        {
            IUIAutomationElementArray items = appElement.GetElement().FindAll(TreeScope.TreeScope_Children, uia.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_MenuBarControlTypeId));
            if (items.Length > 0)
            {
                items = items.GetElement(0).FindAll(TreeScope.TreeScope_Children, uia.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_MenuItemControlTypeId));
                for (int i = 0; i < items.Length; i++)
                {
                    IUIAutomationElement item = items.GetElement(i);
                    if (item.CurrentName == itemName)
                    {
                        return new WindowElement(item);
                    }
                }
            }
        }
        return new WindowElement();
    }

    public WindowElement FindSubmenuItem(WindowElement menuItem, string subitemName)
    {
        if (menuItem.IsValid())
        {
            menuItem.Expand();
            IUIAutomationElementArray items = menuItem.GetElement().FindAll(TreeScope.TreeScope_Descendants, uia.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_MenuItemControlTypeId));
            if (items.Length > 0)
            {
                for (int i = 0; i < items.Length; i++)
                {
                    IUIAutomationElement item = items.GetElement(i);
                    string name = item.CurrentName;
                    if (item.CurrentName == subitemName)
                    {
                        return new WindowElement(item);
                    }
                }
            }
        }
        return new WindowElement();
    }`

When finding the submenu items, I never get any results. I have read other answers that say make sure to use TreeScope_Descendants, which I have done. And then of course answers that say you must Expand the menu, which I have done and can visibly see is working.

WindowElement app = ui.FindProcessNameElement("gMotor2 MAS");
WindowElement fileMenu = ui.FindMenuBarItem(app, "File");
if (fileMenu.Expand())
{
   Console.WriteLine("Expanded file menu");
}
WindowElement exitItem = ui.FindSubmenuItem(fileMenu, "Exit");
exitItem.Invoke();

I expected Invoke() to exit the application. I see the menu expand, but since I never get the submenu items back, the invoke of for the exitItem does nothing.

Answer 1

Here are my classes if anyone wants to just use these as samples. There is a Logging class wrapper on .NET C# logging that you can just ignore. I have added a dependency on IUAutomationClient COM object version 7.2.19041.4355.

using UIAutomationClient;

namespace Example
{
public class WindowElement
{
    private IUIAutomationElement? element;

    private IUIAutomationElement? parent;

    public WindowElement(IUIAutomationElement element, IUIAutomationElement parent)
    {
        this.element = element;
        this.parent = parent;
    }

    public WindowElement(IUIAutomationElement element) 
    { 
        this.element = element;
    }

    public WindowElement()
    {
    }

    public bool IsValid()
    {
        return element != null;
    }

    public IUIAutomationElement GetElement() 
    {
        return element;
    }

    public IUIAutomationElement GetParent()
    {
        return parent;
    }

    public bool Expand()
    {
        if (element != null)
        {
            IUIAutomationExpandCollapsePattern pattern = element.GetCurrentPattern(WindowAutomation.UIA_ExpandCollapsePatternId);
            pattern.Expand();
            return pattern.CurrentExpandCollapseState == ExpandCollapseState.ExpandCollapseState_Expanded;
        }
        return false;
    }

    public bool Invoke()
    {
        if (element != null)
        {
            IUIAutomationInvokePattern pattern = element.GetCurrentPattern(WindowAutomation.UIA_InvokePatternId);
            pattern.Invoke();
            return true;
        }
        return false;
    }

    public bool Select()
    {
        if (element != null)
        {
            IUIAutomationSelectionItemPattern pattern = element.GetCurrentPattern(WindowAutomation.UIA_SelectionItemPatternId);
            pattern.Select();
            return true;
        }
        return false;
    }

    public bool FocusItem(string name)
    {
        if (element != null)
        {
            IUIAutomationItemContainerPattern pattern = element.GetCurrentPattern(WindowAutomation.UIA_ItemContainerPatternId);
            IUIAutomationElement child = pattern.FindItemByProperty(null, WindowAutomation.UIA_NamePropertyId, name);
            child.SetFocus();
            return true;
        }
        return false;
    }
}

public class WindowAutomation
{
    // not sure why I have define these myself since they should be available as constants in the interface.  But I can find
    // no mechanism to access these deinitions from the UI Automation controls.

    // These are the pattern types that we can create on an automation element.  Only certain patterns are valid for
    // certain controls.
    //
    public const int UIA_InvokePatternId = 10000;   // this can be used on any clickable item

    public const int UIA_SelectionPatternId = 10001;

    public const int UIA_ValuePatternId = 10002;

    public const int UIA_RangeValuePatternId = 10003;

    public const int UIA_ScrollPatternId = 10004;

    public const int UIA_ExpandCollapsePatternId = 10005;   // this can be used on menus or any other control that has a dropdown

    public const int UIA_GridPatternId = 10006;

    public const int UIA_GridItemPatternId = 10007;

    public const int UIA_MultipleViewPatternId = 10008;

    public const int UIA_WindowPatternId = 10009;

    public const int UIA_SelectionItemPatternId = 10010;

    public const int UIA_DockPatternId = 10011;

    public const int UIA_TablePatternId = 10012;

    public const int UIA_TableItemPatternId = 10013;

    public const int UIA_TextPatternId = 10014;

    public const int UIA_TogglePatternId = 10015;

    public const int UIA_TransformPatternId = 10016;

    public const int UIA_ScrollItemPatternId = 10017;

    public const int UIA_LegacyIAccessiblePatternId = 10018;

    public const int UIA_ItemContainerPatternId = 10019;

    public const int UIA_VirtualizedItemPatternId = 10020;

    public const int UIA_SynchronizedInputPatternId = 10021;

    public const int UIA_ObjectModelPatternId = 10022;

    public const int UIA_AnnotationPatternId = 10023;

    public const int UIA_TextPattern2Id = 10024;

    public const int UIA_StylesPatternId = 10025;

    public const int UIA_SpreadsheetPatternId = 10026;

    public const int UIA_SpreadsheetItemPatternId = 10027;

    public const int UIA_TransformPattern2Id = 10028;

    public const int UIA_TextChildPatternId = 10029;

    public const int UIA_DragPatternId = 10030;

    public const int UIA_DropTargetPatternId = 10031;

    public const int UIA_TextEditPatternId = 10032;

    public const int UIA_CustomNavigationPatternId = 10033;

    public const int UIA_SelectionPattern2Id = 10034;

    // These are all of the property types that we can use for creating a condition from an automation
    // element.
    //
    public const int UIA_ProcessIdPropertyId = 30002;

    public const int UIA_ControlTypePropertyId = 30003;

    public const int UIA_NamePropertyId = 30005;

    public const int UIA_ClickablePointPropertyId = 30014;

    public const int UIA_ItemTypePropertyId = 300021;


    // These are all of the control types that can be found using UIA_ControlTypePrpoertyId condition or by simply
    // comparing controls from a linear search of the application automation elements.
    //
    public const int UIA_ButtonControlTypeId = 50000;

    public const int UIA_CalendarControlTypeId = 50001;

    public const int UIA_CheckBoxControlTypeId = 50002;

    public const int UIA_ComboBoxControlTypeId = 50003;

    public const int UIA_EditControlTypeId = 50004;

    public const int UIA_HyperlinkControlTypeId = 50005;

    public const int UIA_ImageControlTypeId = 50006;

    public const int UIA_ListItemControlTypeId = 50007;

    public const int UIA_ListControlTypeId = 50008;

    public const int UIA_MenuControlTypeId = 50009;

    public const int UIA_MenuBarControlTypeId = 50010;

    public const int UIA_MenuItemControlTypeId = 50011;

    public const int UIA_ProgressBarControlTypeId = 50012;

    public const int UIA_RadioButtonControlTypeId = 50013;

    public const int UIA_ScrollBarControlTypeId = 50014;

    public const int UIA_SliderControlTypeId = 50015;

    public const int UIA_SpinnerControlTypeId = 50016;

    public const int UIA_StatusBarControlTypeId = 50017;

    public const int UIA_TabControlTypeId = 50018;

    public const int UIA_TabItemControlTypeId = 50019;

    public const int UIA_TextControlTypeId = 50020;

    public const int UIA_ToolBarControlTypeId = 50021;

    public const int UIA_ToolTipControlTypeId = 50022;

    public const int UIA_TreeControlTypeId = 50023;

    public const int UIA_TreeItemControlTypeId = 50024;

    public const int UIA_CustomControlTypeId = 50025;

    public const int UIA_GroupControlTypeId = 50026;

    public const int UIA_ThumbControlTypeId = 50027;

    public const int UIA_DataGridControlTypeId = 50028;

    public const int UIA_DataItemControlTypeId = 50029;

    public const int UIA_DocumentControlTypeId = 50030;

    public const int UIA_SplitButtonControlTypeId = 50031;

    public const int UIA_WindowControlTypeId = 50032;

    public const int UIA_PaneControlTypeId = 50033;

    public const int UIA_HeaderControlTypeId = 50034;

    public const int UIA_HeaderItemControlTypeId = 50035;

    public const int UIA_TableControlTypeId = 50036;

    public const int UIA_TitleBarControlTypeId = 50037;

    public const int UIA_SeparatorControlTypeId = 50038;

    public const int UIA_SemanticZoomControlTypeId = 50039;

    public const int UIA_AppBarControlTypeId = 50040;

    private CUIAutomation uia = new CUIAutomation();

    /// <summary>
    /// Find the window element root for the specific process id
    /// </summary>
    /// <param name="processID">The process id value</param>
    /// <returns>The window element</returns>
    public WindowElement FindProcessIDElement(int processID)
    {
        IUIAutomationElement root = uia.GetRootElement();
        IUIAutomationElementArray children = root.FindAll(TreeScope.TreeScope_Children, uia.CreatePropertyCondition(UIA_ProcessIdPropertyId, processID));
        if (children.Length > 0)
        {
            IUIAutomationElement item = children.GetElement(0);
            IUIAutomationCacheRequest cache = uia.CreateCacheRequest();
            uia.ElementFromHandleBuildCache(item.CurrentNativeWindowHandle, cache);
            return new WindowElement(item);
        }
        return new WindowElement();
    }

    public WindowElement FindProcessNameElement(string processName)
    {
        IUIAutomationElement root = uia.GetRootElement();
        IUIAutomationElementArray children = root.FindAll(TreeScope.TreeScope_Children, uia.CreateTrueCondition());
        for (int i = 0; i < children.Length; i++)
        {
            IUIAutomationElement child = children.GetElement(i);
            string childName = child.CurrentName;
            if (childName.StartsWith(processName))
            {
                return new WindowElement(child);
            }
        }
        return new WindowElement();
    }

    public WindowElement FindMenuBarItem(WindowElement appElement, string itemName) 
    {
        if (appElement.IsValid())
        {
            IUIAutomationElementArray items = appElement.GetElement().FindAll(TreeScope.TreeScope_Children, uia.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_MenuBarControlTypeId));
            if (items.Length > 0)
            {
                IUIAutomationElement menu = items.GetElement(0);
                items = menu.FindAll(TreeScope.TreeScope_Children, uia.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_MenuItemControlTypeId));
                for (int i = 0; i < items.Length; i++)
                {
                    IUIAutomationElement item = items.GetElement(i);
                    Logging.LogDebug("Search menu item: found [{0}] of type [{1}]", item.CurrentName, item.CurrentControlType);
                    if (item.CurrentControlType == UIA_MenuItemControlTypeId)
                    {
                        if (item.CurrentName.StartsWith(itemName))
                        {
                            Logging.LogInfo("Found menu item [{0}]", item.CurrentName);
                            return new WindowElement(item, menu);
                        }
                    }
                }
            }
        }
        return new WindowElement();
    }

    public WindowElement FindMenuBarContainerItem(WindowElement appElement, string itemName)
    {
        if (appElement.IsValid())
        {
            IUIAutomationElementArray items = appElement.GetElement().FindAll(TreeScope.TreeScope_Children, uia.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_MenuControlTypeId));
            if (items.Length > 0)
            {
                for (int i = 0; i < items.Length; i++)
                {
                    IUIAutomationElement item = items.GetElement(0);
                    if (item.CurrentName.StartsWith(itemName))
                    {
                        Logging.LogInfo("Found menu item {0}", item.CurrentName);
                        return new WindowElement(item);
                    }
                }
            }
        }
        return new WindowElement();
    }

    public WindowElement FindSubmenuItem(WindowElement menuItem, string subitemName)
    {
        if (menuItem.IsValid())
        {
            IUIAutomationElementArray items = menuItem.GetElement().FindAll(TreeScope.TreeScope_Descendants, uia.CreateTrueCondition());
            if (items.Length > 0)
            {
                for (int i = 0; i < items.Length; i++)
                {
                    IUIAutomationElement item = items.GetElement(i);
                    string name = item.CurrentName;
                    Logging.LogInfo("Found submenu item: [{0}]", name);
                    if (item.CurrentName.StartsWith(subitemName))
                    {
                        return new WindowElement(item);
                    }
                }
            }
        }
        return new WindowElement();
    }


    public void FindElements(string name)
    {
        IUIAutomationElement root = uia.GetRootElement();
        IUIAutomationElementArray children = root.FindAll(TreeScope.TreeScope_Children, uia.CreateTrueCondition());
        for (int i = 0; i < children.Length; i++)
        {
            IUIAutomationElement child = children.GetElement(i);
            string childName = child.CurrentName;
            Console.WriteLine("Child Window: " + childName + " Process iD: " + child.CurrentProcessId);
            Console.WriteLine("Compare with " + name);
            if (childName.StartsWith(name))
            {
                IUIAutomationElementArray controls = child.FindAll(TreeScope.TreeScope_Children, uia.CreateTrueCondition());
                for (int j = 0; j < controls.Length; j++)
                {
                    IUIAutomationElement control = controls.GetElement(j);
                    string controlName = control.CurrentName;
                    int controlType = control.CurrentControlType;
                    Console.WriteLine("     Control: Name=" + controlName + " Type=" + controlType);
                    if (controlType == UIA_MenuBarControlTypeId)
                    {
                        IUIAutomationElementArray menuItems = control.FindAll(TreeScope.TreeScope_Children, uia.CreateTrueCondition());
                        for (int k = 0; k < menuItems.Length; k++)
                        {
                            IUIAutomationElement menuItem = menuItems.GetElement(k);
                            Console.WriteLine("Menu item: " + menuItem.CurrentName);
                            Console.WriteLine("Menu properties: handle={0} value={1}", menuItem.CurrentNativeWindowHandle, menuItem.GetCurrentPropertyValue(UIA_ClickablePointPropertyId));
                            if (menuItem.CurrentName.StartsWith("File"))
                            {
                                //menuItem.SetFocus();
                                //IUIAutomationExpandCollapsePattern pattern = menuItem.GetCurrentPattern(UIA_ExpandCollapsePatternId);
                                //pattern.Expand();
                                //pattern.Invoke();
                                IUIAutomationElementArray fileItems = menuItem.FindAll(TreeScope.TreeScope_Descendants, uia.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_MenuItemControlTypeId));
                                for (int l = 0; l < fileItems.Length; l++)
                                {
                                    IUIAutomationElement fileItem = fileItems.GetElement(l);
                                    Console.WriteLine("File item: " + fileItem.CurrentName);
                                }

                            }
                        }
                    }
                    else
                    {
                        IUIAutomationElementArray subitems = control.FindAll(TreeScope.TreeScope_Children, uia.CreateTrueCondition());
                        for (int k = 0; k < subitems.Length; k++)
                        {
                            IUIAutomationElement subitem = subitems.GetElement(k);
                            Console.WriteLine("Sub item: " + subitem.CurrentName);
                        }
                    }
                }
            }
        }
    }
}

}

Answer 2

I found the cause of the problem (hope this helps others):

There are two types of menu items at the root level. The first is the menu bar items themselves that can be selected. These are

UIA_MenuItemControlTypeId   (50011)

Then there are the menu bar 'container' items that contain submenu items. These are

UIA_MenuControlTypeId  (50009)

So you find the menu element itself using ...

FindAll(TreeScope.TreeScope_Children, uia.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_MenuItemControlTypeId));

Then you find the submenu items AFTER you Expand() the element you found above using ...

FindAll(TreeScope.TreeScope_Children, uia.CreatePropertyCondition(UIA_ControlTypePropertyId, UIA_MenuControlTypeId));

You can add additional condition to only return the ONE menu item you care about, just be cautious of matching the exact name as there is often trailing info on the menu item names. My code is just doing a search through the returned items so I can log them all and find mistakes in name mismatch etc.

eg. The 'Open' option on my 'File' menu is actually 'Open ... Ctrl+O' not just the string 'Open'.

I hope this helps someone working with this COM library and C#. Keep the type id differences in mind even if you are working in a different language as I am sure the concept is the same.