KiYugadgeter
KiYugadgeter

Reputation: 4012

Cannot find a WebView2 element with UIAutomation

I cannot find the elements of WebView2 within WinForm using UIAutomation.

I want to press the "Login" button to log in to stock trade application. In this application, the user ID and password are stored and automatically filled in, so the user just needs to press the "Login" button. I’m trying to find the login button using UIAutomation, but I can't locate it, and thus cannot click it.

This button appears properly in inspect.exe.

And frustratingly, about once in every 10 tries, for some reason, it can be found, and when that happens, it works.

This login screen is created with WinForm, and the login form inside it is created with WebView2 (confirmed via inspect.exe). I read somewhere that when using FindAll, some elements could be missing, so I also tried using TreeView.RawViewWalker, but I couldn’t find it either.

How can I find it consistently?

Note that this is Japanese software so "Login" is written with Japanese Katakana "ログイン".

Following is code that I tried with two ways that FindAll and TreeViewer.

public class LoginButtonFinder
{
    public static PropertyCondition cond1 = new PropertyCondition(AutomationElement.NameProperty, "ログイン");

    public static void findLoginButton() // Find Methods way
    {
        var ps = Process.GetProcessesByName("KabuS"); // check whether the app's process running.
        if (ps.Length <= 0)
        {
            Console.WriteLine("Kabu Process not found");
            return;
        }
        var kabu_process = ps[0];
        var cond2 = new PropertyCondition(AutomationElement.ProcessIdProperty, kabu_process.Id);
        var cond3 = new AndCondition(cond1, cond2);

        var login_window = AutomationElement.RootElement.FindFirst(TreeScope.Descendants, cond3);
        if (login_window == null)
        {
            Console.WriteLine("Login Window not found");
            return;
        }

        var cond4 = new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Button);
        var cond5 = new AndCondition(cond1, cond4); 
        var login_button = login_window.FindFirst(TreeScope.Descendants, cond5);

        if (login_button != null)
        {
            Console.WriteLine("findLoginButton: Found");
        }
        else
        {
            Console.WriteLine("findLoginButton: Not Found");
        }
    }

    public static void findLoginButtonWithTree() // TreeWalker way
    {
        var ps = Process.GetProcessesByName("KabuS"); //Check whether app's process running
        if (ps.Length <= 0)
        {
            Console.WriteLine("Kabu Process not found");
            return;
        }
        var kabu_process = ps[0];
        var cond2 = new PropertyCondition(AutomationElement.ProcessIdProperty, kabu_process.Id);
        var cond3 = new AndCondition(cond1, cond2);

        var login_window = AutomationElement.RootElement.FindFirst(TreeScope.Descendants, cond3); 
        if (login_window == null)
        {
            Console.WriteLine("Login Window not found"); // login_window(window that has login button can be find constantly.  
            return;
        }
        var found = false;
        var stack = new Stack<AutomationElement>();
        var walker = TreeWalker.RawViewWalker;
        var login_window_first_child = login_window.FindFirst(TreeScope.Children, Condition.TrueCondition);
        if (login_window_first_child != null) {
            stack.Push(login_window_first_child);
        }
        while (stack.Count != 0) {
            var e = stack.Pop();
            var ctrltype = (ControlType)e.GetCurrentPropertyValue(AutomationElement.ControlTypeProperty, true);
            var name = (string)e.GetCurrentPropertyValue(AutomationElement.NameProperty);
            if (ctrltype == ControlType.Button && name == "ログイン") {
                Console.WriteLine("findLoginButtonWithTree: Login Button Found");
                found = true;
                break;
            }
            var nextSibling = walker.GetNextSibling(e);
            if (nextSibling != null) {
                stack.Push(nextSibling);
            }
            var firstChild = walker.GetFirstChild(e);
            if (firstChild != null) {
                stack.Push(firstChild);
            }
        }
        if (!found) {
            Console.WriteLine("findLoginButtonWithTree: Login Button Not Found");
        } 
    }
}

This is a content of the button in inspect.exe:

How found:  Mouse move (1278,660)
    hwnd=0x00000000000204D8 64bit class="Chrome_RenderWidgetHostHWND" style=0x56300000 ex=0x20
Name:   "ログイン"
ControlType:    UIA_ButtonControlTypeId (0xC350)
LocalizedControlType:   "ボタン"
BoundingRectangle:  {l:1057 t:637 r:1377 b:689}
IsEnabled:  true
IsOffscreen:    false
IsKeyboardFocusable:    true
HasKeyboardFocus:   false
ProcessId:  7676
RuntimeId:  [2A.204D8.4.5.1.F]
AutomationId:   ""
FrameworkId:    "Chrome"
ClassName:  "cb69c0ee9 c3386b12d cf1d87149 c76f929a0 _button-login-password"
IsControlElement:   true
IsContentElement:   true
ProviderDescription:    "[pid:7676,providerId:0x0 Main(parent link):Unidentified Provider (unmanaged:msedge.dll)]"
IsPeripheral:   false
AriaRole:   "button"
AriaProperties: "readonly=true;expanded=false;multiline=false;multiselectable=false;required=false;hasactions=false"
LiveSettingProperty:    Off (0)
IsPassword: false
IsRequiredForForm:  false
IsDataValidForForm: true
DescribedBy:    
ControllerForProperty:  
FlowsTo:    
FlowsFrom:  
ClickablePoint: {x:1217 y:663}
Culture:    17
Orientation:    0
OptimizeForVisualContent:   false
AnnotationObjects:  
FullDescription:    ""
IsDialog:   false
LegacyIAccessible.ChildId:  0
LegacyIAccessible.DefaultAction:    "押す"
LegacyIAccessible.Description:  ""
LegacyIAccessible.Help: ""
LegacyIAccessible.KeyboardShortcut: ""
LegacyIAccessible.Name: "ログイン"
LegacyIAccessible.Role: ボタン (0x2B)
LegacyIAccessible.State:    フォーカスできます。 (0x100000)
LegacyIAccessible.Value:    ""
IsAnnotationPatternAvailable:   false
IsDragPatternAvailable: false
IsDockPatternAvailable: false
IsDropTargetPatternAvailable:   false
IsExpandCollapsePatternAvailable:   false
IsGridItemPatternAvailable: false
IsGridPatternAvailable: false
IsInvokePatternAvailable:   true
IsItemContainerPatternAvailable:    false
IsLegacyIAccessiblePatternAvailable:    true
IsMultipleViewPatternAvailable: false
IsObjectModelPatternAvailable:  false
IsRangeValuePatternAvailable:   false
IsScrollItemPatternAvailable:   true
IsScrollPatternAvailable:   false
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    false
IsSpreadsheetItemPatternAvailable:  false
IsSpreadsheetPatternAvailable:  false
IsStylesPatternAvailable:   false
IsSynchronizedInputPatternAvailable:    false
IsTableItemPatternAvailable:    false
IsTablePatternAvailable:    false
IsTextChildPatternAvailable:    true
IsTextEditPatternAvailable: false
IsTextPatternAvailable: false
IsTextPattern2Available:    false
IsTogglePatternAvailable:   false
IsTransformPatternAvailable:    false
IsTransform2PatternAvailable:   false
IsValuePatternAvailable:    false
IsVirtualizedItemPatternAvailable:  false
IsWindowPatternAvailable:   false
IsCustomNavigationPatternAvailable: false
IsSelectionPattern2Available:   false
FirstChild: [null]
LastChild:  [null]
Next:   [null]
Previous:   "" グループ
Other Props:    Object has no additional properties
Children:   Container has no children
Ancestors:  "" グループ
    "" グループ
    "" グループ
    "" グループ
    "" メイン
    "ログイン画面|三菱UFJ eスマート証券" ドキュメント
    "Chrome Legacy Window" ウィンドウ
    "" 領域
    "" 領域
    "" 領域
    "" 領域
    "" 領域
    "" 領域
    "ログイン画面|三菱UFJ eスマート証券 - Web コンテンツ" 領域
    "ログイン画面|三菱UFJ eスマート証券" ウィンドウ
    "" ウィンドウ
    "" ウィンドウ
    "ログイン" ウィンドウ
    "デスクトップ 1" ウィンドウ
    [ No Parent ]

Upvotes: 0

Views: 23

Answers (0)

Related Questions