Windows MIDI streaming and SysEx

Question

I am using the ancient Windows Multimedia API (midiXyz functions from WinMM.dll) from C#.

After opening a Midi Out device/port in non-streaming mode (midiOutOpen), sending SysEx with (midiOutLongMsg) works fine.

After opening a Midi Out device/port in streaming mode (midiStreamOpen), sending SysEx with midiOutLongMsg does not work.

Instead, midiOutLongMsg fails with error MMSYSERR_NOTSUPPORTED (= 8). The error text is: "This function is not supported. Use the Capabilities function to determine which functions and messages the driver supports."

However, according to MSDN, (midiOutLongMsg) should also work with stream handles. Jeff Glatt's excellent MIDI information pages also claim, that SysEx and streaming can be used together (see end of page).

Sending buffered SysEx messages by enqueueing them with (midiStreamOut) midiStreamOut works fine. However, I need/want to send SysEx directly using midiOutLongMsg, too.

I have already checked out various open source Midi libraries (managed as well as unmanaged), several Midi driver sources and even the WinMM.dll sources of WINE, but could not find any hints what I am doing wrong.

To reproduce my problem in the smallest possible code, I ripped out all callback, unprepare, cleanup and release stuff, and condensed several classes in just one function. The following code opens the first Midi device/port and tries to send a "GM Mode On" SysEx message:

Update 2014 January 12th: Please see code version 3 below !

public static class MidiTest { // version 1 - x86/32 bit only

  public static void Test () {
    int moID = 0; // midi out device/port ID
    int moHdl; // midi out device/port handle
#if !true
    // SysEx via midiOutLongMsg works
    Chk (WinMM.midiOutOpen (out moHdl, moID, null, 0, 0)); // open midi out in non-stream mode
#else
    // SysEx via midiOutLongMsg fails
    Chk (WinMM.midiStreamOpen (out moHdl, ref moID, 1, null, 0, 0)); // open midi out in stream mode
#endif
    byte [] sx = { 0xF0, 0x7E, 0x7F, 0x09, 0x01, 0xF7 }; // GM On sysex
    int shdr = Marshal.SizeOf (typeof (MidiHdr)); // hdr size
    var mhdr = new MidiHdr (); // allocate managed hdr
    mhdr.bufferLength = mhdr.bytesRecorded = sx.Length; // length of message bytes
    mhdr.data = Marshal.AllocHGlobal (mhdr.bufferLength); // allocate native message bytes
    Marshal.Copy (sx, 0, mhdr.data, mhdr.bufferLength); // copy message bytes from managed to native memory
    IntPtr nhdr = Marshal.AllocHGlobal (shdr); // allocate native hdr
    Marshal.StructureToPtr (mhdr, nhdr, false); // copy managed hdr to native hdr
    Chk (WinMM.midiOutPrepareHeader (moHdl, nhdr, shdr)); // prepare native hdr
    Chk (WinMM.midiOutLongMsg (moHdl, nhdr, shdr)); // send native message bytes
  } // Test

  static void Chk (int f) {
    if (0 == f) return;
    var sb = new StringBuilder (256); // MAXERRORLENGTH
    var s = 0 == WMM.midiOutGetErrorText (f, sb, sb.Capacity) ? sb.ToString () : String.Format ("MIDI Error {0}.", f);
    System.Diagnostics.Trace.WriteLine (s);
  }

  [StructLayout (LayoutKind.Sequential)]
  internal struct MidiHdr { // sending long MIDI messages requires a header
    public IntPtr data; // native pointer to message bytes, allocated on native heap
    public int bufferLength; // length of buffer 'data'
    public int bytesRecorded; // actual amount of data in buffer 'data'
    public int user; // custom user data
    public int flags; // information flags about buffer
    public IntPtr next; // reserved
    public int reserved; // reserved
    public int offset; // buffer offset on callback
    [MarshalAs (UnmanagedType.ByValArray, SizeConst = 4)]
    public int[] reservedArray; // reserved
  } // struct MidiHdr

  internal sealed class WinMM { // native MIDI calls from WinMM.dll
    public delegate void CB (int hdl, int msg, int inst, int p1, int p2); // callback
    [DllImport ("winmm.dll")] public static extern int midiStreamOpen (out int hdl, ref int devID, int reserved, CB proc, int inst, int flags);
    [DllImport ("winmm.dll")] public static extern int midiOutOpen (out int hdl, int devID, CB proc, int inst, int flags);
    [DllImport ("winmm.dll")] public static extern int midiOutPrepareHeader (int hdl, IntPtr pHdr, int sHdr);
    [DllImport ("winmm.dll")] public static extern int midiOutLongMsg (int hdl, IntPtr pHdr, int sHdr);
    [DllImport ("winmm.dll")] public static extern int midiOutGetErrorText (int err, StringBuilder msg, int sMsg);
  } // class WinMM

} // class MidiTest

Question 1: Is it possible to send SysEx via midiOutLongMsg when the Midi device/port has been opened in streaming mode (midiStreamOpen) ?

Question 2: If yes, any idea what I am missing ?

Question 3: I did not found many sources using MIDI in streaming mode. So, if you know some open source library using MIDI output in streaming mode, please throw me a link so I can compare ..

Thanks

Update (2013 Oct 19):

Hi CL,

thanks for your answer. Yes, maybe someone at Microsoft messed something up in maintaining WinMM.dll - but I think the probability that I am missing something is higher, because

a) There is an old manual "Windows NT DDK - Multimedia Drivers" (available here), that describes the WinMM stuff in more detail than the current MSDN pages. Page 56 shows a diagram with WinMM.dll as an intermediate layer between the application and the MIDI/Audio driver(s). WinMM's main job is to pass the MIDI data from the application to one of the MIDI/Audio drivers. When the MIDI driver is a port driver for an external keyboard/synthesizer/tone generator, WinMM cant change the MIDI data that much. Page 94 describes the driver message MODM_LONGDATA and its parameters - its pretty much identical to the parameters of midiOutLongMsg. That limits the opportunities to mess something up inside WinMM.dll.

b) The code path in WinMM.dll from midiOutLongMsg being called to calling the driver with MODM_LONGDATA works fine after midiOutOpen, but not after midiStreamOpen. The result code is MMSYSERR_NOTSUPPORTED - that tells me that I am getting slapped by some sanity check at the beginning of the code path in WinMM.dll, like

if (whatever_weird_condition) return MMSYSERR_NOTSUPPORTED;

And that whatever_weird_condition is most likely about something I should have done, but have not done ..

c) If the MIDI driver doesnt support streaming output by itself (its optional), WinMM does the translation from buffered/enqueued output (midiStreamOut) to the simpler non-streaming driver calls (which are not optional). None of the 8 MIDI drivers on my machine supports streaming itself, all rely on WinMM to do it. Streaming short and long messages works fine.

d) My test code behaves exactly the same on Windows 7 and Windows XP. If Microsoft messed something up, the bug must have been made quite a while ago (before XP). And I am the either the first one to find it (after years), or everybody else kept it top secret and ungoogleable.

e) My test code behaves exactly the same with all 8 midi drivers on my machine. That tells me, that it is most likely not a driver issue.

f) Years of debugging have taught me, that if something doesnt work as it should, the problem is most likely on my side of the screen .. ;-P

Answer 1

Here is a 4th version of the test code, this time in native C++, to make sure that my problem is not related to interop and managed code stuff. The problem (error code 8) persists:

#include <stdio.h>
#include <tchar.h>
#include <windows.h>
#include <MMSystem.h>

#pragma comment ( lib, "winmm.lib" )

static void Chk (UINT r) {
  static _TCHAR errmsg[256];
  if (!r) return;
  UINT rr = midiOutGetErrorText (r, errmsg, sizeof (errmsg));
  printf ("MIDI Error %d: '%S'\n", r, errmsg);
}

int _tmain (int argc, _TCHAR* argv[]) {
  UINT moID = 0; // first midi out device/port ID
  DWORD_PTR inst = NULL; // no instance
  DWORD_PTR clbk = NULL; // no callback
  DWORD flgs = CALLBACK_NULL; // flags, no callback
#if 0
  // SysEx via midiOutLongMsg works
  HMIDIOUT hmo = 0; // midi out device/port handle
  Chk (midiOutOpen (&hmo, moID, clbk, inst, flgs)); // open midi out in non-stream mode
#else
  // SysEx via midiOutLongMsg fails
  HMIDISTRM hms = 0; // midi out device/port handle
  Chk (midiStreamOpen (&hms, &moID, 1, clbk, inst, flgs)); // open midi out in stream mode
  HMIDIOUT hmo = (HMIDIOUT) hms;
#endif
  Chk (midiOutShortMsg (hmo, 0x00404090)); // note on
  Sleep (200); // ms
  Chk (midiOutShortMsg (hmo, 0x00004090)); // note off
  static unsigned char sx [] = { 0xF0, 0x7E, 0x7F, 0x09, 0x01, 0xF7 }; // GM On sysex
  static MIDIHDR mhdr; // midi header describes long msg
  UINT shdr = sizeof (mhdr);
  memset (&mhdr, 0, shdr); // clear header
  mhdr.lpData = (LPSTR) sx; // point to sysex
  mhdr.dwBufferLength = mhdr.dwBytesRecorded = sizeof (sx); // length of message bytes
  Chk (midiOutPrepareHeader (hmo, &mhdr, shdr)); // prepare hdr
  UINT r = midiOutLongMsg (hmo, &mhdr, shdr); // send message bytes
  Chk (r);
  // unprepare header, close etc. omitted ...
  return 0;
}

Answer 2

I don't know if this helps but if you wrap your sysex in a MidiEvent structure such as this (r - relative tick, s - stream ID, e - event code, d - data, p - pad):

//MidiEvent - r, r, r, r, s, s, s, s, e, e, e,   e,   d,   d,   d, d, d,   d, p, p
byte[] sx = { 9, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 128, 240, 126, 127, 9, 1, 247, 0, 0 }; // GM On sysex

and add these lines after you prepare the header:

Chk(WinMM.midiStreamOut(moHdl, nhdr, shdr));
int r = WinMM.midiOutLongMsg(moHdl, nhdr, shdr);
Chk(r); // send native message bytes
Chk(WinMM.midiStreamRestart(moHdl));

You will see the midiStreamOut sysex go out the port after a slight delay from the relative tick field of the MidiEvent. You will still get an error for the midiOutLongMessage but the error code is 65 or "Cannot perform this operation while media data is still playing. Reset the device, or wait until the data is finished playing."

I don't know if that is helpful but it is different from the error code 8.

Answer 3

Update: I apologize for not getting back sooner. I have been swamped at work. Yes, you are right it is failing now. Maybe I was up too late that night but I cannot understand how it was working or even if it was. I also need to send out sysex in stream mode but my app has not done that yet only in non-stream mode (midiOutOpen). I will keep looking at it to see if I can figure out a way around it. Do you have to use the sysex master volume or can you use the CC:7 volume control? Of course, this will not help for sysex but short messages can get through in stream mode. Oh, thanks for the update, I was also getting the code to compile and run in x86 or x64 (AnyCPU).

Original Message: I do not know if you are still interested but I think the code below may answer your questions. I put your old code under the comment //PREVIOUS CODE and the new code under the comment //NEW CODE.

Additionally, for x86, the size of the header should not include the data. I know the size is 0x40 for x86 but I am still trying to figure out the best way to code this so if you have any ideas let me know.

I just figured this out myself for another application so I have not fleshed it all out yet but I ran this code and it seems to work for you. I love streaming mode in this old dll. It is very precise and if you are using double buffering you can make it real time... you can also send short messages out in stream mode the same way you do for midiout.

Hint: the code below is version 2, in part compatible with x86/x64 32/64-bit. (MillKa)

using System;

using System.Collections.Generic;

using System.Text;

using System.Runtime.InteropServices;

using System.Diagnostics;

public static class MidiTest
{

    public static void Test() 
    {
        int moID = 0; // midi out device/port ID

        //PREVIOUS CODE
        //int moHdl; // midi out device/port handle
        //NEW CODE
        IntPtr moHdl = IntPtr.Zero;

#if !true
    // SysEx via midiOutLongMsg works
    Chk (WinMM.midiOutOpen (out moHdl, moID, null, 0, 0)); // open midi out in non-stream mode
#else
        // SysEx via midiOutLongMsg fails
        //PREVIOUS CODE
        //Chk(WinMM.midiStreamOpen(out moHdl, ref moID, 1, null, 0, 0)); // open midi out in stream mode
        //NEW CODE
        IntPtr instance = IntPtr.Zero;
        Chk(WinMM.midiStreamOpen(out moHdl, ref moID, 1, null, instance, 0)); // open midi out in stream mode

#endif
        byte[] sx = { 0xF0, 0x7E, 0x7F, 0x09, 0x01, 0xF7 }; // GM On sysex

        //PREVIOUS CODE
        //int shdr = Marshal.SizeOf(typeof(MidiHdr)); // hdr size
        //NEW CODE
        int shdr = 0x40; // hdr size

        var mhdr = new MidiHdr(); // allocate managed hdr
        mhdr.bufferLength = mhdr.bytesRecorded = sx.Length; // length of message bytes
        mhdr.data = Marshal.AllocHGlobal(mhdr.bufferLength); // allocate native message bytes
        Marshal.Copy(sx, 0, mhdr.data, mhdr.bufferLength); // copy message bytes from managed to native memory
        IntPtr nhdr = Marshal.AllocHGlobal(shdr); // allocate native hdr
        Marshal.StructureToPtr(mhdr, nhdr, false); // copy managed hdr to native hdr
        Chk(WinMM.midiOutPrepareHeader(moHdl, nhdr, shdr)); // prepare native hdr
        Chk(WinMM.midiOutLongMsg(moHdl, nhdr, shdr)); // send native message bytes
    } // Test

    static void Chk(int f)
    {
        if (0 == f) return;
        var sb = new StringBuilder(256); // MAXERRORLENGTH
        var s = 0 == WinMM.midiOutGetErrorText(f, sb, sb.Capacity) ? sb.ToString() : String.Format("MIDI Error {0}.", f);
        System.Diagnostics.Trace.WriteLine(s);
    }

    [StructLayout(LayoutKind.Sequential)]
    internal struct MidiHdr
    { // sending long MIDI messages requires a header
        public IntPtr data; // native pointer to message bytes, allocated on native heap
        public int bufferLength; // length of buffer 'data'
        public int bytesRecorded; // actual amount of data in buffer 'data'
        public int user; // custom user data
        public int flags; // information flags about buffer
        public IntPtr next; // reserved
        public int reserved; // reserved
        public int offset; // buffer offset on callback
        [MarshalAs(UnmanagedType.ByValArray, SizeConst = 4)]
        public int[] reservedArray; // reserved
    } // struct MidiHdr

    internal sealed class WinMM
    { // native MIDI calls from WinMM.dll
        public delegate void CB(int hdl, int msg, int inst, int p1, int p2); // callback

        //PREVIOUS CODE
        //[DllImport("winmm.dll")]
        //public static extern int midiStreamOpen(out int hdl, ref int devID, int reserved, CB proc, int inst, int flags);
        //[DllImport("winmm.dll")]
        //public static extern int midiOutOpen(out int hdl, int devID, CB proc, int inst, int flags);
        //[DllImport("winmm.dll")]
        //public static extern int midiOutPrepareHeader(int hdl, IntPtr pHdr, int sHdr);
        //[DllImport("winmm.dll")]
        //public static extern int midiOutLongMsg(int hdl, IntPtr pHdr, int sHdr);
        //[DllImport("winmm.dll")]
        //public static extern int midiOutGetErrorText(int err, StringBuilder msg, int sMsg);

        //NEW CODE
        #region winmm declarations
        [DllImport("winmm.dll")]
        public static extern int midiOutPrepareHeader(IntPtr handle,
            IntPtr headerPtr, int sizeOfMidiHeader);
        [DllImport("winmm.dll")]
        public static extern int midiOutUnprepareHeader(IntPtr handle,
            IntPtr headerPtr, int sizeOfMidiHeader);
        [DllImport("winmm.dll")]
        public static extern int midiOutOpen(out IntPtr handle, int deviceID,
            CB proc, IntPtr instance, int flags);
        [DllImport("winmm.dll")]
        public static extern int midiOutGetErrorText(int errCode,
            StringBuilder message, int sizeOfMessage);
        [DllImport("winmm.dll")]
        public static extern int midiOutClose(IntPtr handle);
        [DllImport("winmm.dll")]
        public static extern int midiStreamOpen(out IntPtr handle, ref int deviceID, int reserved,
            CB proc, IntPtr instance, uint flag);
        [DllImport("winmm.dll")]
        public static extern int midiStreamClose(IntPtr handle);
        [DllImport("winmm.dll")]
        public static extern int midiStreamOut(IntPtr handle, IntPtr headerPtr, int sizeOfMidiHeader);
        [DllImport("winmm.dll")]
        public static extern int midiOutLongMsg(IntPtr handle,
            IntPtr headerPtr, int sizeOfMidiHeader);
        #endregion

    } // class WinMM

} // class MidiTest

Answer 4

Here is version 3 of the test code:

public static class MidiTest { // version 3 - x68/x64 32/64-bit compatible

  public static void Test () {
    int moID = 0; // midi out device/port ID
    IntPtr moHdl = IntPtr.Zero;

#if !true
    // SysEx via midiOutLongMsg works
    Chk (WinMM.midiOutOpen (out moHdl, moID, null, 0, 0)); // open midi out in non-stream mode
#else
    // SysEx via midiOutLongMsg fails
    IntPtr instance = IntPtr.Zero;
    Chk (WinMM.midiStreamOpen (out moHdl, ref moID, 1, null, instance, 0)); // open midi out in stream mode
#endif
    byte[] sx = { 0xF0, 0x7E, 0x7F, 0x09, 0x01, 0xF7 }; // GM On sysex

    int shdr = Marshal.SizeOf(typeof(MidiHdr)); // hdr size
    IntPtr x = Marshal.OffsetOf (typeof (MidiHdr), "data");   // ptr; size: 4/8, offset: 0
    x = Marshal.OffsetOf (typeof (MidiHdr), "bufferLength");  // int; size: 4  , offset: 4/8
    x = Marshal.OffsetOf (typeof (MidiHdr), "bytesRecorded"); // int; size: 4  , offset: 8/12
    x = Marshal.OffsetOf (typeof (MidiHdr), "user");          // ptr; size: 4/8, offset: 12/16
    x = Marshal.OffsetOf (typeof (MidiHdr), "flags");         // int; size: 4  , offset: 16/24; followed by 4 byte padding
    x = Marshal.OffsetOf (typeof (MidiHdr), "next");          // ptr; size: 4/8, offset: 20/32
    x = Marshal.OffsetOf (typeof (MidiHdr), "reserved");      // ptr; size: 4/8, offset: 24/40
    x = Marshal.OffsetOf (typeof (MidiHdr), "offset");        // int; size: 4  , offset: 28/48; followed by 4 byte padding
    x = Marshal.OffsetOf (typeof (MidiHdr), "reservedArray"); // ptr; size: 4/8 x 8 = 32/64, offset: 32/56
    // total size: 64/120
    var mhdr = new MidiHdr (); // allocate managed hdr
    mhdr.bufferLength = mhdr.bytesRecorded = sx.Length; // length of message bytes
    mhdr.data = Marshal.AllocHGlobal (mhdr.bufferLength); // allocate native message bytes
    Marshal.Copy (sx, 0, mhdr.data, mhdr.bufferLength); // copy message bytes from managed to native memory
    IntPtr nhdr = Marshal.AllocHGlobal (shdr); // allocate native hdr
    Marshal.StructureToPtr (mhdr, nhdr, false); // copy managed hdr to native hdr
    Chk (WinMM.midiOutPrepareHeader (moHdl, nhdr, shdr)); // prepare native hdr
    int r = WinMM.midiOutLongMsg (moHdl, nhdr, shdr); // send native message bytes
    Chk (r); // send native message bytes
  } // Test

  static void Chk (int f) {
    if (0 == f) return;
    var sb = new StringBuilder (256); // MAXERRORLENGTH
    var s = 0 == WinMM.midiOutGetErrorText (f, sb, sb.Capacity) ? sb.ToString () : String.Format ("MIDI Error {0}.", f);
    System.Diagnostics.Trace.WriteLine (s);
  }

  [StructLayout (LayoutKind.Sequential)]
  internal struct MidiHdr { // sending long MIDI messages requires a header
    public IntPtr data; // native pointer to message bytes, allocated on native heap
    public int bufferLength; // length of buffer 'data'
    public int bytesRecorded; // actual amount of data in buffer 'data'
    public IntPtr user; // custom user data
    public int flags; // information flags about buffer
    public IntPtr next; // reserved
    public IntPtr reserved; // reserved
    public int offset; // buffer offset on callback
    [MarshalAs (UnmanagedType.ByValArray, SizeConst = 8)]
    public IntPtr[] reservedArray; // reserved
  } // struct MidiHdr

  internal sealed class WinMM { // native MIDI calls from WinMM.dll
    public delegate void CB (IntPtr hdl, int msg, IntPtr inst, int p1, int p2); // callback
    [DllImport ("winmm.dll")] public static extern int midiStreamOpen (out IntPtr hdl, ref int devID, int reserved, CB proc, IntPtr inst, uint flags);
    [DllImport ("winmm.dll")] public static extern int midiOutOpen (out IntPtr hdl, int devID, CB proc, IntPtr instance, int flags);
    [DllImport ("winmm.dll")] public static extern int midiOutPrepareHeader (IntPtr hdl, IntPtr pHdr, int sHdr);
    [DllImport ("winmm.dll")] public static extern int midiOutLongMsg (IntPtr hdl, IntPtr pHdr, int sHdr);
    [DllImport ("winmm.dll")] public static extern int midiOutGetErrorText (int err, StringBuilder msg, int sMsg);
  } // class WinMM

} // class MidiTest

Like Mark's version 2 above, it uses IntPtr where needed to be compatible with 32 AND 64 bit.

I also modified the declaration of the MidiHdr struct:

• pointers are declared as IntPtr because of 32/64 bit
• the size of the array at the end of the header was wrong (4), it is correct now (8)

With the changed declaration, Marshal.SizeOf now calculates the correct size:

• 0x40 = 64 on 32 bit
• 0x78 = 120 on 64 bit

To align the pointer fields next and reservedArray, the compiler adds 4 byte padding after the int fields flags and offset. Field sizes and offset are shown in the comments.

Unfortunately, my version 3 and Mark's version 2 both still have the original problem: midiOutLongMsg still returns error code 8. I tested the code compiled as x86 (32 bit) as well as x64 (64 bit) on Windows 7 Ultimate, 64 bit, SP1. The 32 bit version was also tested on Windows XP Pro, SP3, 32 bit. Same result on all platforms.

If someone wonders why I need to send Sysex messages while streaming: If a midi file is played using WinMM streaming and the user changes the volume slider, I need to send the new volume via Sysex - without stopping the streaming.

Answer 5

Almost nobody uses MIDI streams.

Nowadays, the midi* functions are not longer implemented by the hardware vendors' drivers but by Microsoft's MM WDM compatibility driver. It appears that this detail was overlooked in the rewrite.