CODEWITHSUNDEEP
macoscocoaaccessibility
Bri Bri
Bri Bri

Reputation: 2278

How to create an AXUIElementRef from an NSView or NSWindow?

Concerning the macOS Accessibility API, is there anyway to create an AXUIElementRef that corresponds to either an NSView or an NSWindow?

There appears to have been a way of doing this back in the days of Carbon using AXUIElementCreateWithHIObjectAndIdentifier but that function isn't available anymore.

The only method I'm aware of is to use the Accessibility API to recursively search the entire hierarchy of UI elements of your application looking for one that matches the NSView or NSWindow. But in addition to being an onerous solution, it's not even guaranteed to succeed as there might not be a way to positively correspond an AXUIElementRef and a Cocoa object by just using the available attributes of the AXUIElementRef.

I am willing to consider undocumented APIs that help accomplish this.

Upvotes: 3

Views: 1215

Answers (2)

Bri Bri
Bri Bri

Reputation: 2278

Here we are, just over eight years later and I think I've finally found a solution. The trick is to use this undocumented function that luckily stills works in modern macOS:

extern "C" AXError _AXUIElementGetWindow(AXUIElementRef, CGWindowID *out);

This way we can use the Accessibility API to search through the list of windows of an application and compare their window IDs with the ID of an NSWindow to find a match.

Here's a function that demonstrates how to do this:

AXUIElementRef copyWindowWithPIDAndWindowID(pid_t pid, CGWindowID targetWindowId)
{
    AXUIElementRef result = nullptr;
    AXUIElementRef app = AXUIElementCreateApplication(pid);
    
    if (!app) {
        return nullptr;
    }
    
    CFArrayRef windows;
    AXError error = AXUIElementCopyAttributeValue(app, kAXWindowsAttribute, (const void **)&windows);
    
    if (error != kAXErrorSuccess || !windows) {
        CFRelease(app);
        return nullptr;
    }
    
    for(long i = 0; i < CFArrayGetCount(windows); ++i) {
        CGWindowID windowId;
        AXUIElementRef window = (AXUIElementRef)CFArrayGetValueAtIndex(windows, i);
        error = _AXUIElementGetWindow(window, &windowId);
        
        if (error != kAXErrorSuccess || targetWindowId == 0) {
            qDebug() << "Error: couldn't get CGWindowID from AXUIElementRef. Error code:" << error;
            continue;
        }
        
        if (windowId == targetWindowId) {
            CFRetain(window);
            result = window;
            break;
        }
    }
    
    CFRelease(app);
    CFRelease(windows);
    
    return result;
}

You could use it with an NSWindow like this:

NSWindow *window = ...;
copyWindowWithPIDAndWindowID(getpid(), window.windowNumber);

This function also allows getting an AXUIElementRef to a window from another application.

Upvotes: 0

Elist
Elist

Reputation: 5533

I found the way to do the same thing in iOS.

I know this isn't a direct answer to your question, but I'll try to explain what I have done to find it in iOS, and hopefully you'll be able to do the same in macOS. Also, this might be useful for other readers...

I started by guessing that the process itself is creating the AXUIElementRef, so it has to create them when I request accessibility attributes that have AXUIElementRef values, such as kAXUIElementAttributeChildren.

I then created an app, and dlsym'ed _AXUIElementCreateAppElementWithPid(int pid), calling it with [[NSProcessInfo processInfo] processIdentifier]. I received the root AXUIElementRef, which I then passed into AXError AXUIElementCopyMultipleAttributeValues(AXUIElementRef element, CFArrayRef attributes, AXCopyMultipleAttributeOptions options, CFArrayRef _Nullable *values), requesting kAXUIElementAttributeChildren, and it worked (should be run on main thread)!

I started debugging the AXUIElementCopyMultipleAttributeValues call carefully into the assembly code, which went pretty much like that (this is very pseudo-code, off course...):

// serialize the arguments for MIG call
if (axUIElementRef->pid == self pid) {
    // that's good that we are calling our own process, we can easily keep debugging!
    _MIGXAAXUIElementCopyMultipleAttributeValues(serialized arguments) {
         // Get the original element from the AXUIElementRef:
         UIView* view = _AXElementForAXUIElementUniqueId(id);
         [view accessibilityAttributeValue:kAXUIElementAttributeChildren] {
              [view _accessibilityUserTestingChildren] {
                   // since this is the UIApplication element, it just gets the windows:
                   NSArray*<UIWindow*> winArr = [(UIApplication*)view _accessibilityWindows];
                   // for each value in result, convert to AX value:
                   ...
                   AXConvertOutgoingValue(winArr) {
                       // For each UIView, convert to AXUIElementRef:
                       AXUIElementRef e = _AXCreateAXUIElementWithElement(winArr[i]);
                   }
              }
         }
    }
} else {
    // Do the same only if we are entitled, and outside our process
}

So, in iOS, you simply call AXUIElementRef _AXCreateAXUIElementWithElement(UIView*); to convert from UI to accessibility element, and UIView* _AXElementForAXUIElementUniqueId(AXUIElementRefGetUniqueID(AXUIElementRef)); in the opposite direction.

All symbols are from AXRuntime.framework.

In mac, you'll need to link against ApplicationServices.framework and try something similar.

Hope this helps...

Upvotes: 3

Related Questions