Convert Microphone Input into Keyboard Input

Question

I have a keyboard pedal (as in a piano keyboard) that I've plugged into my microphone jack. I want to be able to translate the input into keystrokes (e.g. Press "V" on my COMPUTER keyboard whenever I step on the pedal). I know I can use Java's AWT Robot to perform the keystrokes but I'm having trouble processing the microphone input (I have no experience with audio processing).

This is the signal I get stepping and release the pedal:

This seems like it might be a pretty simple task for an experienced user, does anyone out there know how I might do it?

So far I've basically copy-pasted code from around the place to get this:

I'm using the javax.sound API to read the microphone input as bytes. I'm trying to detect the jump in amplitude when I step on the pedal... The bytes are converted to shorts and the value is compared against an arbitrarily high number. It seems to work but after one pedal depression, the robot just keeps pressing the key (value > 32000) forever.

        while (true) {
            // Read the next chunk of data from the TargetDataLine.
            numBytesRead = microphone.read(data, 0, data.length);
            // Save this chunk of data.
            out.write(data, 0, numBytesRead);

            byte[] bytes = out.toByteArray();
            short[] shorts = new short[bytes.length/2];
            // to turn bytes to shorts as either big endian or little endian. 
            ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts);

            for (short s : shorts) {
                int value = Math.abs(s);
                if (value > 32000)
                {
                    robot.keyPress(KeyEvent.VK_V);
                    robot.keyRelease(KeyEvent.VK_V);
                    break;
                } else {
                    //robot.keyRelease(KeyEvent.VK_V);
                }
            }
        }

Edit:

It was going on and on forever because I wasn't clearing the ByteArrayOutputStream buffer, so I was continually reading the same set of bytes plus the new ones as time went by. out.reset() solved that for me.

My issue now is about the input I read from the pedal, I can't properly interpret a press or a release if I press and release the pedal within a short time frame.

The red circle shows what I get when the pedal is depressed (and held), the black rectangle shows when the pedal is released.

As you can see, when it's depressed, it drops down then quickly increases before gradually going back to 0 again. When it's released, it shoots up and then quickly drops down before going back to 0 again.

The method I'm using now to differentiate between the two is to only register a press/release when there's a large difference between two frames/intervals. As per the graphs, I make it a press when it's a negative value and a release when it's a positive value.

The issue I'm having is that when I release the pedal when the signal is on the upswing (or when I press the pedal whilst the signal is on the downswing of a release), there won't be enough of a difference between the two frames for me to use this method.

This might help with visualizing the issue:

I don't know how to get a robust way to detect a press/release.

Here's my fixed code for reading the input, if anyone is interested in trying the same thing (I'm using this pedal by the way: http://www.amazon.co.uk/Cherub-WTB-004-Keyboard-Sustain-Pedal/dp/B000UDVV6E)

import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

import javax.sound.sampled.*;

import java.awt.AWTException;
import java.awt.Robot;
import java.awt.event.KeyEvent;

public class PedalToKeyboard {

    public static void main(String[] args) {

        AudioFormat format = new AudioFormat(8000.0f, 16, 1, true, true);
        Robot robot = null;

        try {
            robot = new Robot();

        } catch (AWTException e) {
                e.printStackTrace();
        }

        try {
            TargetDataLine microphone = AudioSystem.getTargetDataLine(format);          
            System.out.println(microphone);

            microphone.open(format);

            ByteArrayOutputStream out = new ByteArrayOutputStream();
            int numBytesRead;
            byte[] data = new byte[microphone.getBufferSize()/5];

            // Begin audio capture.
            microphone.start();

            boolean keep_going = true;
            boolean keyPressed = false;         
            short previousShort = 0;
            while (keep_going) {

                // Read the next chunk of data from the TargetDataLine.
                numBytesRead = microphone.read(data, 0, data.length);

                // Reset the buffer (get rid of previous shit)
                out.reset();

                // Save this chunk of data.
                out.write(data, 0, numBytesRead);

                byte[] bytes = out.toByteArray();
                short[] shorts = new short[bytes.length/2];

                // to turn bytes to shorts as either big endian or little endian. 
                ByteBuffer.wrap(bytes).order(ByteOrder.BIG_ENDIAN).asShortBuffer().get(shorts);

                for (short s : shorts) {

                    // Check if descending or ascending (pedal press is descending, release is ascending)
                    if (s < 0) { // descending                  
                        // make sure drop is large instantaneous drop
                        if (Math.abs(Math.abs(previousShort) - Math.abs(s)) > 10000 && s < -32700) {
                            if (!keyPressed) {
                                keyPressed = true;
                                //robot.keyPress(KeyEvent.VK_V);
                                System.out.println("Pressed: " + s);
                                break;
                            }
                        }
                    } else if (s > 0) { // ascending
                        // make sure increase is large instantaneous increase
                        if (Math.abs(Math.abs(previousShort) - Math.abs(s)) > 10000 && s > 32700) {
                            if (keyPressed) {
                                keyPressed = false;
                                //robot.keyRelease(KeyEvent.VK_V);
                                System.out.println("Released: " + s + "\n");
                                break;
                            }
                        }
                    }

                    previousShort = s;
                }
            }    

        } catch (LineUnavailableException e) {
            e.printStackTrace();
        }
    }
}

Answer 1

Alright I've actually tweaked the values a bit and fixed some of my logic (I was wrongly applying Math.abs when comparing the difference).

This is working quite well for me now, anyone out there with MIDI pedals can try tweaking the parameters for your own use.

package pedal2keyboard;

import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

import javax.sound.sampled.*;

import java.awt.AWTException;
import java.awt.Robot;
import java.awt.event.KeyEvent;

/***
 * Author: Dois Koh
 * Date: 27th October 2015
 * 
 * Gets your microphone signal and you can go do whatever you want with it.
 * Right now, it takes signals from my Cherub WTB-004 Keyboard Sustain Pedal, plugged into
 * my microphone jack, and converts it into key presses (holds down V when depressed,
 * releases V when released)
 */
public class PedalToKeyboard {

    // Robot for performing keyboard actions (pressing V)
    public static Robot robot = null;

    // Currently 8KHz, 16 bit signal (2 bytes), single channel, signed (+ and -) and BIG ENDIAN format
    public static AudioFormat format = new AudioFormat(8000.0f, 16, 1, true, true);

    public static TargetDataLine microphone = null;
    public static boolean pedalPressed = false;

    public static void main(String[] args) {

        try {
            // Initialize robot for later use
            robot = new Robot();

            // Retrieve the line to from which to read in the audio signal
            microphone = AudioSystem.getTargetDataLine(format);

            // Open the line in the specified format -
            // Currently 8KHz, 16 bit signal (2 bytes), single channel, signed (+ and -) and BIG ENDIAN format      
            microphone.open(new AudioFormat(8000.0f, 16, 1, true, true));

            ByteArrayOutputStream out = new ByteArrayOutputStream();
            byte[] data = new byte[microphone.getBufferSize()/8];

            // Begin audio capture.
            microphone.start();

            int numBytesRead = 0;
            short previousShort = 0;

            // Continue until program is manually terminated
            while (true) {

                // Read the next chunk of data from the TargetDataLine.
                numBytesRead = microphone.read(data, 0, data.length);

                // Reset the buffer (get rid of previous data)
                out.reset();

                // Save this chunk of data.
                out.write(data, 0, numBytesRead);

                byte[] bytes = out.toByteArray();
                short[] shorts = new short[bytes.length/2];

                // to turn bytes to shorts as either big endian or little endian. 
                ByteBuffer.wrap(bytes).order(ByteOrder.BIG_ENDIAN).asShortBuffer().get(shorts);

                // Iterate through retrieved 16 bit data (shorts)
                for (short s : shorts) {

                    // Check if descending or ascending (pedal press is descending, release is ascending)
                    if (s < 0) { // descending                  
                        // make sure drop is large instantaneous drop
                        if (Math.abs(previousShort - s) > 200 && s < -32700) {
                            if (!pedalPressed) {
                                PedalPressedAction();
                                previousShort = s;
                                break;
                            }
                        }
                    } else if (s > 0) { // ascending
                        // make sure increase is large instantaneous increase
                        if (Math.abs(previousShort - s) > 200 && s > 32700) {
                            if (pedalPressed) {
                                PedalReleasedAction();
                                previousShort = s;
                                break;
                            }
                        }
                    }

                    previousShort = s;
                }
            }    

        } catch (LineUnavailableException | AWTException e) {
            e.printStackTrace();
        } finally {
            if (microphone != null)
                microphone.close();
        }
    }

    /***
     * The action to perform when the pedal is depressed
     */
    public static void PedalPressedAction() {
        pedalPressed = true;
        robot.keyPress(KeyEvent.VK_V);
    }

    /***
     * The action to perform when the pedal is released
     */
    public static void PedalReleasedAction(){
        pedalPressed = false;
        robot.keyRelease(KeyEvent.VK_V);        
    }
}

Answer 2

If you look at the graph you attached you can see that the value stays above 32000 for quite a long period of time - maybe around 100ms or so. Depending on your sample rate this can translate into quite a large number of keypresses. For example at 44.1kHz this would be 4410 of them. You need to only simulate a keypress when you initially cross the threshold. To do so you'll need to keep track of some state. You can play with the values for PRESS_THRESHOLD and RELEASE_THRESHOLD but start with something like 32000 and 31000. The reason for not setting them the same is to prevent false key presses if the signal is glitchy right around the edge.

for (short s : shorts) {
    int value = Math.abs(s);
    switch (state)
    {
    case State.pressed:
        if (value < RELEASE_THRESHOLD)
        {
            state = State.released;
        }
        break;
    case State.released:
        if (value > PRESS_THRESHOLD)
        {           
            robot.keyPress(KeyEvent.VK_V);
            robot.keyRelease(KeyEvent.VK_V);
            state = State.pressed;
        }
        break;
    }
}