Cause of serial port transmitting bad data and WriteFile return wrong number of bytes written?

Question

Bugfix update: As of Jun, 2013 FTDI did acknowledge to me that the bug was real. They have since released a new version of their driver (2.8.30.0, dated 2013-July-12) that fixes the problem. The driver made it through WHQL around the first of August, 2013 and is available via Windows Update at this time.

I've re-tested running the same test code and am not able to reproduce the problem with the new driver, so at the moment the fix seems to be 'upgrade the driver'.

The original question: I've got an 8 port USB-serial device (from VsCOM) that is based on the FTDI FT2232D chip. When I transmit at certain settings from one of the ports, AND I use the hardware handshaking to stop and start the data flow from the other end, I get two symptoms:

1) The output data sometimes becomes garbage. There will be NUL characters, and pretty much any random thing you can think of.

2) The WriteFile call will sometimes return a number of bytes GREATER than the number I asked it to write. That's not a typo. I ask for 30 bytes to be transmitted and the number of bytes sent comes back 8192 (and yes, I do clear the number sent to 0 before I make the call).

Relevant facts: Using FTDI drivers 2.8.24.0, which is the latest as of today. Serial port settings are 19200, 7 data bits, odd parity, 1 stop bit. I get this same behavior with another FTDI based serial device, this time a single port one. I get the same behavior with another 8 port device of the same type. I do NOT get this behavior when transmitting on the built-in serial ports (COM1). I have a very simple 'Writer' program that just transmits continuously and a very simple 'Toggler' program that toggles RTS once per second. Together these seem to trigger the issue within 60 seconds. I have put an issue into the manufacturer of the device, but they've not yet had much time to respond. Compiler is mingw32, the one included with the Qt installer for Qt 4.8.1 (gcc 4.4.0)

I'd like to know first off, if there's anything that anyone can think of that I could possibly do to trigger this behavior. I can't conceive of anything, but there's always things I don't know.

Secondly, I've attached the Writer and Toggler test programs. If anyone can spot some issue that might trigger the program, I'd love to hear about it. I have a lot of trouble thinking that there is a driver bug (especially from something as mature as the FTDI chip), but the circumstances force me to think that there's at least SOME driver involvement. At the least, no matter what I do to it, it shouldn't be returning a number of bytes written greater than what I asked it to write.

Writer program:

#include <iostream>
#include <string>

using std::cerr;
using std::endl;

#include <stdio.h>
#include <windows.h>



int main(int argc, char **argv)
{
    cerr << "COM Writer, ctrl-c to end" << endl;

    if (argc != 2) {
        cerr << "Please specify a COM port for parameter 2";
        return 1;
    }

    char fixedbuf[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";


    std::string portName = "\\\\.\\";
    portName += argv[1];

    cerr << "Transmitting on port " << portName << endl;

    HANDLE ph =  CreateFileA( portName.c_str(),
                              GENERIC_READ | GENERIC_WRITE,
                              0,      //  must be opened with exclusive-access
                              NULL,   //  default security attributes
                              OPEN_EXISTING, //  must use OPEN_EXISTING
                              0,      //  overlapped I/O
                              NULL ); //  hTemplate must be NULL for comm devices

    if (ph == INVALID_HANDLE_VALUE) {
        cerr << "CreateFile " << portName <<  " failed, error " << GetLastError() << endl;
        return 1;
    }


    COMMCONFIG  ccfg;
    DWORD ccfgSize = sizeof(COMMCONFIG);
    ccfg.dwSize = ccfgSize;

    GetCommConfig(ph, &ccfg, &ccfgSize);
    GetCommState(ph, &(ccfg.dcb));

    ccfg.dcb.fBinary=TRUE;
    ccfg.dcb.fInX=FALSE;
    ccfg.dcb.fOutX=FALSE;
    ccfg.dcb.fAbortOnError=FALSE;
    ccfg.dcb.fNull=FALSE;

    // Camino is 19200 7-O-1
    ccfg.dcb.BaudRate = 19200;
    ccfg.dcb.Parity = ODDPARITY;
    ccfg.dcb.fParity = TRUE;
    ccfg.dcb.ByteSize = 7;
    ccfg.dcb.StopBits = ONESTOPBIT;

    // HW flow control
    ccfg.dcb.fOutxCtsFlow=TRUE;
    ccfg.dcb.fRtsControl=RTS_CONTROL_HANDSHAKE;
    ccfg.dcb.fInX=FALSE;
    ccfg.dcb.fOutX=FALSE;

    COMMTIMEOUTS ctimeout;
    DWORD tout = 10;// 10 ms
    ctimeout.ReadIntervalTimeout = tout;
    ctimeout.ReadTotalTimeoutConstant = tout;
    ctimeout.ReadTotalTimeoutMultiplier = 0;
    ctimeout.WriteTotalTimeoutMultiplier = tout;
    ctimeout.WriteTotalTimeoutConstant = 0;


    SetCommConfig(ph, &ccfg, sizeof(COMMCONFIG));
    SetCommTimeouts(ph, &ctimeout);

    DWORD nwrite = 1;
    for(;;) {
        nwrite++;
        if (nwrite > 30) nwrite = 1;

        DWORD nwritten = 0;
        if (!WriteFile(ph, fixedbuf, nwrite, &nwritten, NULL)) {
            cerr << "f" << endl;
        }

        if ((nwritten != 0) && (nwritten != nwrite)) {
            cerr << "nwrite: " << nwrite << " written: " << nwritten << endl;
        }
    }


    return 0;
}

Toggler program:

#include <iostream>
#include <string>

using std::cerr;
using std::endl;

#include <stdio.h>
#include <windows.h>



int main(int argc, char **argv)
{
    cerr << "COM Toggler, ctrl-c to end" << endl;
    cerr << "Flips the RTS line every second." << endl;

    if (argc != 2) {
        cerr << "Please specify a COM port for parameter 2";
        return 1;
    }


    std::string portName = "\\\\.\\";
    portName += argv[1];

    cerr << "Toggling RTS on port " << portName << endl;

    HANDLE ph =  CreateFileA( portName.c_str(),
                              GENERIC_READ | GENERIC_WRITE,
                              0,      //  must be opened with exclusive-access
                              NULL,   //  default security attributes
                              OPEN_EXISTING, //  must use OPEN_EXISTING
                              0,      //  overlapped I/O
                              NULL ); //  hTemplate must be NULL for comm devices

    if (ph == INVALID_HANDLE_VALUE) {
        cerr << "CreateFile " << portName <<  " failed, error " << GetLastError() << endl;
        return 1;
    }


    COMMCONFIG  ccfg;
    DWORD ccfgSize = sizeof(COMMCONFIG);
    ccfg.dwSize = ccfgSize;

    GetCommConfig(ph, &ccfg, &ccfgSize);
    GetCommState(ph, &(ccfg.dcb));

    ccfg.dcb.fBinary=TRUE;
    ccfg.dcb.fInX=FALSE;
    ccfg.dcb.fOutX=FALSE;
    ccfg.dcb.fAbortOnError=FALSE;
    ccfg.dcb.fNull=FALSE;

    // Camino is 19200 7-O-1
    ccfg.dcb.BaudRate = 19200;
    ccfg.dcb.Parity = ODDPARITY;
    ccfg.dcb.fParity = TRUE;
    ccfg.dcb.ByteSize = 7;
    ccfg.dcb.StopBits = ONESTOPBIT;

    // no flow control (so we can do manually)
    ccfg.dcb.fOutxCtsFlow=FALSE;
    ccfg.dcb.fRtsControl=RTS_CONTROL_DISABLE;
    ccfg.dcb.fInX=FALSE;
    ccfg.dcb.fOutX=FALSE;

    COMMTIMEOUTS ctimeout;
    DWORD tout = 10;// 10 ms
    ctimeout.ReadIntervalTimeout = tout;
    ctimeout.ReadTotalTimeoutConstant = tout;
    ctimeout.ReadTotalTimeoutMultiplier = 0;
    ctimeout.WriteTotalTimeoutMultiplier = tout;
    ctimeout.WriteTotalTimeoutConstant = 0;


    SetCommConfig(ph, &ccfg, sizeof(COMMCONFIG));
    SetCommTimeouts(ph, &ctimeout);

    bool rts = true;// true for set
    for(;;) {
        if (rts)
            EscapeCommFunction(ph, SETRTS);
        else
            EscapeCommFunction(ph, CLRRTS);

        rts = !rts;
        Sleep(1000);// 1 sec wait.
    }


    return 0;
}

Answer 1

If anyone else is still seeing this -- update your FTDI driver to 2.8.30.0 or later, as this is caused by a driver bug in earlier versions of the FTDI driver.

Answer 2

I don't have a good answer yet from FTDI, but I've got the following suggestions for anyone dealing with this issue:

1) Consider switching to a non-FTDI usb-serial converter. This is what my company did, but certainly this isn't an option for everyone (we're putting the chip in our own product). We're using Silicon Labs chips now, but I think there are one or two other vendors as well.

2) Per Hans Passant in the comments - reconsider the use of RTS/CTS signalling. If the writes don't fail due to blocking, then you shouldn't trigger this bug.

3) Set all writes to infinite timeout. Again, no fail due to blocking, no triggering of the bug. This may not be appropriate for all applications, of course.

Note that if pursuing strategy #3, if Overlapped IO is used for writes, then CancelIo and it's newer cousin CancelIoEx could be used to kill off the writes if necessary. I have NOT tried doing so, but I suspect that such cancels might also result in triggering this bug. If they were only used when closing the port anyway, then it might be you could get away with it, even if they do trigger the bug.