Trigger drop event on another application in OSX

Question

I use a bit of proprietary software for DJing (Native Instruments Traktor). If you're not familiar with this kind of thing, just think of it as a glorified iTunes: it browses and plays audio files.

I want to build a custom file browser application for this, but there isn't any kind of API. It does, however, allow audio files to be dragged and dropped from the file system, which is a great start.

The nature of the file browser I'm designing means I don't want to actually have to drag and drop things - I just want to click a component in my application, and have the same effect. So I'm looking at ways to programmatically trigger a drop event on the other application, from my file browser application.

My platform of choice is Python with PyQt, but I'm beginning to feel I might have to go a bit lower-level. Haven't done a huge amount of C#/Java though so that could be a bit of a learning curve (I have done a lot of ANSI C but that's probably too low level...)

This is how far I've got:

• I've made a really simple PyQt application
• I can create QDrag object when a QLabel in my application is dragged
• I can attach all the correct MIME data to represent the audio file to this
• So if I drag and drop that QLabel from my application into Traktor, it recognises the audio file and plays it - good times

So now I need to cut out the middle-man and, on click, package up my MIME data and make Traktor think I've dragged and dropped it onto it.

I've also done some delving in the OSX developers' docs, specifically this stuff, which describes the sequence of messages that are passed to the target application (drop destination).

This all makes sense, but I'm on the verge of dropping down to C#/Java to try and mimic these messages, which sounds like rabbit hole I'd rather not venture down if I can avoid it.

So, before I do...

1. Is this even possible? Or am I going to hit some kind of cross-app security barriers etc? (drop destinations only accepting messages directly from the OS or something)
2. If it is, is there an easier way to do it? Ideally with PyQt/Tkinter/wxPython...?

I know I could do this with click automation, but I can imagine that being really unreliable, would be massively reliant on the positions of windows, etc.

Thanks in advance!

Answer 1

I know this is an old post, but I'm in the midst of developing my of Traktor Suggestions program. I've got it 95% there and now I just needed a way to be able to allow the user to select a file and then click either "Load to Deck A" or "Load to Deck B". As you know Traktor will accept dragged files. But I wanted to automate this as when your DJing the less you touch the mouse the better.

I'm also super interested in what you've been working on.

It took me a bit to figure out, but I realized I needed to create a paste board. Since I wasn't dealing with an image and just needed to supply the file path (as a NSString... I could have also used a NSURL, but straight path seemed easiest). They were so many methods for creating the pasteboard and the dragging session and setting up the "dragging image" etc. Then I came across the simplest form which was to use the simple NSView function (needs to be places inside a Mouse Down function). And dragFilePath variable has already been set. So within my Custom NSView I have this code. You also need to have a NSImageView as a subView of Custom NSView. In order for this "quick" function to work.

The self dragFile.... creates an instant pasteboard item and dragging session all will out multiple lines of code.

- (void)mouseDown:(NSEvent*)theEvent {
 NSLog(@"DRAGnDROP VIEW mouseDown happened");
NSLog(@"DRAGnDROP VIEW mouseDown dragFilePath is: %@", dragFilePath);
[self dragFile:dragFilePath fromRect:(self.bounds) slideBack:YES event:theEvent];
}

I have two buttons that then trigger the CGEvents. I have the buttons run functions in Applescript. The Applescript functions trigger the mouse down, start the drake, switch over to Traktor, then move the mouse to either Deck A or Deck B, then release.

AppleScript Functions:

on loadTraktorDeckA:sender
    deckDetailControllerDelegate's loadForDrag:me
    delay 0.5
    tell application "Traktor"
        activate
    end tell
    deckDetailControllerDelegate's loadForReleaseA:me
end loadTraktorDeckA:

on loadTraktorDeckB:sender
    deckDetailControllerDelegate's loadForDrag:me
    delay 0.5
    tell application "Traktor"
        activate
    end tell
    deckDetailControllerDelegate's loadForReleaseB:me
end loadTraktorDeckB:

and in the custom NSView these are the CG Mouse Events that get called:

-(void)loadForDrag:(id)sender {
NSLog(@"mouse left drag called");
/* create a new Quartz mouse event.
 * @source : CGEventSourceRef
 * @mouseType : CGEventType
 * @mouseCursorPosition : CGPoint
 * @mouseButton : CGMouseButton
 */
CGEventSourceStateID kCGEventSourceStatePrivate = -1;
CGEventSourceRef loadDragEventRef = CGEventSourceCreate(kCGEventSourceStatePrivate);

CGPoint startPoint = CGPointMake(880.0, 770.0);
CGPoint movePoint1 = CGPointMake(610.0, 320.0);
CGEventRef leftDownEvent = CGEventCreateMouseEvent(loadDragEventRef, kCGEventLeftMouseDown, startPoint, 1);
CGEventRef leftDragEvent1 = CGEventCreateMouseEvent(loadDragEventRef, kCGEventLeftMouseDragged, startPoint, 0);
CGEventRef leftDragEvent2 = CGEventCreateMouseEvent(loadDragEventRef, kCGEventLeftMouseDragged, movePoint1, 0);
/* post a Quartz event into the event stream at a specified location.
 * @tap : CGEventTapLocation
 * @event : CGEventRef
 */
CGEventPost(kCGHIDEventTap, leftDragEvent2);

CGEventSourceSetLocalEventsSuppressionInterval(loadDragEventRef, 2);
CGEventPost(kCGHIDEventTap, leftDownEvent);
CGEventPost(kCGHIDEventTap, leftDragEvent1);
CGEventPost(kCGHIDEventTap, leftDragEvent2);

/**
 * release a Quartz event
 */
    // CFRelease(leftDragEvent);}

-(void)loadForReleaseA:(id)sender {
NSLog(@"mouse left Up called DECK A");
 CGPoint movePoint1 = CGPointMake(610.0, 320.0);
CGPoint movePointRelease = CGPointMake(220.0, 320.0);

CGEventRef leftDragEvent2 = CGEventCreateMouseEvent(NULL, kCGEventLeftMouseDragged, movePoint1, 0);
CGEventPost(kCGHIDEventTap, leftDragEvent2);

CGEventRef leftClickUpEvent = CGEventCreateMouseEvent(NULL, kCGEventLeftMouseUp,movePointRelease, 0);
CGEventPost(kCGHIDEventTap, leftClickUpEvent);

/** release a Quartz event
 */
CFRelease(leftClickUpEvent);}


-(void)loadForReleaseB:(id)sender {
NSLog(@"mouse left Up called DECK B");
CGPoint movePoint1 = CGPointMake(610.0, 320.0);
CGPoint movePointRelease = CGPointMake(1000.0, 320.0);

CGEventRef leftDragEvent2 = CGEventCreateMouseEvent(NULL, kCGEventLeftMouseDragged, movePoint1, 0);
CGEventPost(kCGHIDEventTap, leftDragEvent2);

CGEventRef leftClickUpEvent = CGEventCreateMouseEvent(NULL, kCGEventLeftMouseUp,movePointRelease, 0);
CGEventPost(kCGHIDEventTap, leftClickUpEvent);

CFRelease(leftClickUpEvent);}

and here is the full Custom DragNDropView Class

DragNDropView.h

    //
//  DragNDropView.h
//  DJK-Tel Traktor Suggestions
//
//
//

#import <Cocoa/Cocoa.h>
#import <Foundation/Foundation.h>
#import <AppKit/AppKit.h>

@interface DragNDropView : NSView <NSDraggingSource, NSDraggingDestination, NSImageDelegate, NSApplicationDelegate>

    //NSPasteboardItemDataProvider

@property (nonatomic, strong) NSString* dragFilePath;
@property (nonatomic, strong) NSURL* dragFilePathURL;

- (id)initWithCoder:(NSCoder *)coder;
- (id)initWithFrame:(NSRect)frameRect;
- (void)mouseDown:(NSEvent*)theEvent;
-(IBAction)loadForDrag:(id)sender;
-(IBAction)loadForReleaseA:(id)sender;
-(IBAction)loadForReleaseB:(id)sender;
- (BOOL)shouldDelayWindowOrderingForEvent:(NSEvent *)event;

@end

DragNDropView.m

    //  DragNDropView.m
    //  DJK-Tel Traktor Suggestions
    //
    //
    //

#import "DragNDropView.h"

@implementation DragNDropView


@synthesize dragFilePath;
@synthesize dragFilePathURL;

- (id)initWithCoder:(NSCoder *)coder
{
    /*------------------------------------------------------
     Init method called for Interface Builder objects
     --------------------------------------------------------*/
    self=[super initWithCoder:coder];
    if ( self ) {

        NSLog(@"DRAGnDROP VIEW INIT WITH CODER happened");
            //[self registerForDraggedTypes:[NSArray arrayWithObjects:@"NSFilenamesPboardType",@"NSURLPboardType",nil]];
        [self initView];
    }
    return self;
}
- (id)initWithFrame:(NSRect)frame
{
    self = [super initWithFrame:frame];
    if (self) {
        NSLog(@"DRAGnDROP VIEW INIT WITH FRAME happened");
        [self initView];
    }
    return self;
}


- (void)setFrame:(NSRect)frameRect
{
    [super setFrame:frameRect];
}

- (void) initView
{
    NSLog(@"DRAGnDROP VIEW Init View");

    dragFilePath = @"";
    dragFilePathURL = nil;
}

- (void)drawRect:(NSRect)dirtyRect {
    [super drawRect:dirtyRect];
        // Drawing code here.
}


#pragma mark - Destination Operations

- (void)mouseDown:(NSEvent*)theEvent {
    NSLog(@"DRAGnDROP VIEW mouseDown happened");
    NSLog(@"DRAGnDROP VIEW mouseDown dragFilePath is: %@", dragFilePath);
    [self dragFile:dragFilePath fromRect:(self.bounds) slideBack:YES event:theEvent];
}

- (BOOL)shouldDelayWindowOrderingForEvent:(NSEvent *)event {
    return YES;
}

- (void)mouseDragged:(NSEvent *)event {
}


- (BOOL)performDragOperation:(id <NSDraggingInfo>)sender {
    NSPasteboard *pboard = [sender draggingPasteboard];
        //NSLog(@"DRAGnDROP VIEW performDragOperation pboard is: %@", pboard);

    if ( [[pboard types] containsObject:NSFilenamesPboardType] ) {
        NSArray *files = [pboard propertyListForType:NSFilenamesPboardType];

        if ([files count] == 1) {
            dragFilePath = files[1];
            return YES;
        }
    }
    else if ( [[pboard types] containsObject:NSURLPboardType] ) {
        dragFilePathURL = [NSURL URLFromPasteboard:pboard];
        NSLog(@"DRAGnDROP VIEW performDragOperation dragFilePathURL is: %@", dragFilePathURL);
        return YES;
    }
    return NO;
}

-(void)loadForDrag:(id)sender {
    NSLog(@"mouse left drag called");
    /* create a new Quartz mouse event.
     * @source : CGEventSourceRef
     * @mouseType : CGEventType
     * @mouseCursorPosition : CGPoint
     * @mouseButton : CGMouseButton
     */
    CGEventSourceStateID kCGEventSourceStatePrivate = -1;
    CGEventSourceRef loadDragEventRef = CGEventSourceCreate(kCGEventSourceStatePrivate);

    CGPoint startPoint = CGPointMake(880.0, 770.0);
    CGPoint movePoint1 = CGPointMake(610.0, 320.0);
    CGEventRef leftDownEvent = CGEventCreateMouseEvent(loadDragEventRef, kCGEventLeftMouseDown, startPoint, 1);
    CGEventRef leftDragEvent1 = CGEventCreateMouseEvent(loadDragEventRef, kCGEventLeftMouseDragged, startPoint, 0);
    CGEventRef leftDragEvent2 = CGEventCreateMouseEvent(loadDragEventRef, kCGEventLeftMouseDragged, movePoint1, 0);
    /* post a Quartz event into the event stream at a specified location.
     * @tap : CGEventTapLocation
     * @event : CGEventRef
     */
    CGEventPost(kCGHIDEventTap, leftDragEvent2);

    CGEventSourceSetLocalEventsSuppressionInterval(loadDragEventRef, 2);
    CGEventPost(kCGHIDEventTap, leftDownEvent);
    CGEventPost(kCGHIDEventTap, leftDragEvent1);
    CGEventPost(kCGHIDEventTap, leftDragEvent2);

    /**
     * release a Quartz event
     */
        // CFRelease(leftDragEvent);
}

-(void)loadForReleaseA:(id)sender {
    NSLog(@"mouse left Up called DECK A");
    CGPoint movePoint1 = CGPointMake(610.0, 320.0);
    CGPoint movePointRelease = CGPointMake(220.0, 320.0);

    CGEventRef leftDragEvent2 = CGEventCreateMouseEvent(NULL, kCGEventLeftMouseDragged, movePoint1, 0);
    CGEventPost(kCGHIDEventTap, leftDragEvent2);

    CGEventRef leftClickUpEvent = CGEventCreateMouseEvent(NULL, kCGEventLeftMouseUp,movePointRelease, 0);
    CGEventPost(kCGHIDEventTap, leftClickUpEvent);

    CFRelease(leftClickUpEvent);
}

-(void)loadForReleaseB:(id)sender {
    NSLog(@"mouse left Up called DECK B");
    CGPoint movePoint1 = CGPointMake(610.0, 320.0);
    CGPoint movePointRelease = CGPointMake(1000.0, 320.0);

    CGEventRef leftDragEvent2 = CGEventCreateMouseEvent(NULL, kCGEventLeftMouseDragged, movePoint1, 0);
    CGEventPost(kCGHIDEventTap, leftDragEvent2);

    CGEventRef leftClickUpEvent = CGEventCreateMouseEvent(NULL, kCGEventLeftMouseUp,movePointRelease, 0);
    CGEventPost(kCGHIDEventTap, leftClickUpEvent);

    CFRelease(leftClickUpEvent);
}

#pragma mark - Source Operations


- (NSDragOperation)draggingSession:(NSDraggingSession *)session sourceOperationMaskForDraggingContext:(NSDraggingContext)context
{
    /*------------------------------------------------------
     NSDraggingSource protocol method.  Returns the types of operations allowed in a certain context.
     --------------------------------------------------------*/
    switch (context) {
        case NSDraggingContextOutsideApplication:
            return NSDragOperationCopy;

                //by using this fall through pattern, we will remain compatible if the contexts get more precise in the future.
        case NSDraggingContextWithinApplication:
        default:
            return NSDragOperationCopy;
                //return NSDragOperationNone;
            break;
    }
}

- (BOOL)acceptsFirstMouse:(NSEvent *)event
{
    /*------------------------------------------------------
     accept activation click as click in window
     --------------------------------------------------------*/
        //so source doesn't have to be the active window
    return NO;
}


@end

Answer 2

Progress! While I still don't know what event is delivered to the application, I do know where the file data is stored: it's on the drag pasteboard! Try dragging a file somewhere and then running this:

#include <stdlib.h>
#import <Foundation/Foundation.h>
#import <AppKit/NSPasteboard.h>

int main(int argc, char **argv) {
    (void)argc, (void)argv;
    NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
    NSPasteboard *dragPasteboard = [NSPasteboard pasteboardWithName:NSDragPboard];
    NSLog(@"%@", [dragPasteboard stringForType:(NSString *)kUTTypeFileURL]);
    [pool drain];
    return EXIT_SUCCESS;
}

Answer 3

Not tried this, but things like CGEventCreateMouseEvent and CGEventPostToPSN might be helpful. CGEvent.h and CGRemoteOperation.h

I'm also wondering if the target application might respond to apple events - if it does you could create apple events and send those to it which would be cleaner. I'd try running AppleScript Editor or Automator and open dictionary on the app in question to see if it has a dictionary of events that might do what you want.

luck.