Recieving extra byte in UART comms

Question

I have been involved in a project where I need to extract some data from a device and display it on a PC to be checked. The device I am recieving data from sends a string which includes the device ID, current mode, temperature reading and battery reading. For ease, these are separated by a comma. An example of a string would be:

01,03,66661242,28

so this would bean the device ID is 1, the mode is mode 3, a temperature reading of 36.6 (this is in ASCII little endian format), and battery level is 4.0V (sent in ASCII and divided by 10)

I have no control over the format the data is sent

I am using an STM32F091RC Nucleo board for this and the code I have is:

#include "mbed.h"

Serial pc(PA_2, PA_3);
Serial Unit (PA_9, PA_10, 9600);                                                // 9600 baud rate - no parity - 1 stop bit

//Input pins
DigitalIn START(PB_8, PullUp);

void GetData();
void CheckData();

char Data[100];

int deviceId;
int Mode;
float TempReading;
float battReading;

unsigned char Ascii2Hex (unsigned char data)
{
    if (data > '9')data += 9;   // add offset if value > 9
    return (data &= 0x0F);
}

unsigned char Ascii2Char(unsigned char Offset)
{
    unsigned char Ans;
    Ans = Ascii2Hex(Data[Offset]);
    Ans = Ans<<4;
    Ans += Ascii2Hex(Data[Offset+1]);
    return(Ans);
}

float Ascii2Float(unsigned char Offset)
{
    float Bob;
    unsigned char Ans;
    Ans = Ascii2Hex(Data[Offset+6]);
    Ans = Ans<<4;
    Ans += Ascii2Hex(Data[Offset+7]);
    ((unsigned char*)&Bob)[3]= Ans;
    Ans = Ascii2Hex(Data[Offset+4]);
    Ans = Ans<<4;
    Ans += Ascii2Hex(Data[Offset+5]);
    ((unsigned char*)&Bob)[2]= Ans;
    Ans = Ascii2Hex(Data[Offset+2]);
    Ans = Ans<<4;
    Ans += Ascii2Hex(Data[Offset+3]);
    ((unsigned char*)&Bob)[1]= Ans;
    Ans = Ascii2Hex(Data[Offset]);
    Ans = Ans<<4;
    Ans += Ascii2Hex(Data[Offset+1]);
    ((unsigned char*)&Bob)[0]= Ans;
    return(Bob);
}

void DecodeString()
{
    char x;

    //numbers in brackets is where the data starts in the string
    deviceId = Ascii2Char(0);
    Mode = Ascii2Char(3);
    TempReading = Ascii2Float(6);
    x = Ascii2Char(15);
    battReading = (float)x/10;
    GetData();
}

void GetData()
{
    Unit.scanf("%s,",Data);  // scan the incoming data on the RX line
    pc.printf("%s,\n\r",Data);
    pc.printf("Device ID = %i\n\r", deviceId);
    pc.printf("Mode = %i\n\r", Mode);
    pc.printf("Temp = %.1f\n\r", TempReading);
    pc.printf("Bat = %.1f\n\n\r", battReading);
}

int main()
{
    while(1) {
        if(START == 0) {
            wait(0.1);
            DecodeString();
        }
    }
}

When I first start up and press the button to get the data, the string I recieve has an extra 0 at the front: 001,03,66661242,28

This then means the data is incorrect as the data has shifted, however, if I press it again, it then gives the correct string but the printed data is incorrect Another press and everything works fine and will continue working until the Nucleo board is reset. An example of the recieved string and displayed data from my serial monitor is:

001,03,33331342,28,
Device ID = 0
Mode = 0
Temp = 0.0
Bat = 0.0

01,03,CDCC1242,28,
Device ID = 0
Mode = 192
Temp = 0.0
Bat = 19.4

01,03,CDCC1242,28,
Device ID = 1
Mode = 3
Temp = 36.7
Bat = 4.0

I am not an expert coder, I am very much a beginner. The bit of code that decodes the string was given to me by the engineer who designed the device that sends the data string. I have for assistance but because of working from home and people being very busy with other things, this isn't a pressing issue so help is limited.

I have tried adding some delays in various places (such as after the original scanf and before printing) and I have also tried the scanf function 3 times just as an experiment to see if I can bypass the incorrect data but none of these have helped. I have tried using different UART pins (the STM32F091RC 64 pin device has 6 available) but I still get the same result. I have also changed the data byte length from 100 to 17 as that is the amount I am expecting to recieve but it still makes no difference.

I have made sure that all devices are sharing a common GND and double checked all the hardware connections.

All I want to do is recieve the correct data first time and display the correct result first time, but I can't seem to get it working.

EDIT

I have now tried adding in an extra few lines. I am using strlen to count the number of bytes in the string. If it is more than 17, I then retry. This has eliminated the first issue, but the first set of decoded data is still displayed incorrectly:

String Length = 18

String Length = 17

01,03,66661242,28,
Device ID = 0
Mode = 192
Temp = 0.0
Bat = 19.4

String Length = 17

01,03,66661242,28,
Device ID = 1
Mode = 3
Temp = 36.6
Bat = 4.0

Is there any way first to make sure the data is decoded correctly first time, or that the data is read correctly first time instead of needing a workaround?

Answer 1

This answers your problem:

The way your code is written, GetData() is printing the “Device ID”, “Mode”, “Temp” and “Bat” data from the previously acquired data, not from the most recently acquired data. That’s why your first set of data is all-zeros; on the first time through, all those variables still contain their original uninitialized values, which since they are statically allocated “global data”, is all-zeros. And the second time through, it’s printing the results you obtained from the first reading, which gives you the wrong value for “Device ID” because of the extra zero at the start of the bytestream. Finally, the third time through, it prints the data you obtained from the second reading, which was good.

If you just rearrange some of your code, it will print the data based on the most recent sample. I haven't tried compiling & running it, but this seems to me to be a likely good rewrite of it, which combines your DecodeString() and GetData() functions into one function:

void DecodeString()
{
    char x;

    // scan the incoming data on the RX line
    Unit.scanf("%s,",Data);

    // translate the data from ASCII into int/float native types
    // numbers in brackets is where the data starts in the string
    deviceId = Ascii2Char(0);
    Mode = Ascii2Char(3);
    TempReading = Ascii2Float(6);
    x = Ascii2Char(15);
    battReading = (float)x/10;

    // print the original unprocessed input string
    pc.printf("%s,\n\r",Data);

    // print what we translated
    pc.printf("Device ID = %i\n\r", deviceId);
    pc.printf("Mode = %i\n\r", Mode);
    pc.printf("Temp = %.1f\n\r", TempReading);
    pc.printf("Bat = %.1f\n\n\r", battReading);
}

You may also get better results if, at startup, you flush your incoming data stream (read & discard any existing buffered garbage), which may resolve your issue with the extra character on your first read. But if there's a startup race condition between both ends of your comms link, you may find that (perhaps occasionally) your receiver begins processing the first sample in the middle of the packet's characters, and perhaps that flush operation would have discarded the first part of the packet. While an initial flush is a good idea, it's even more important that you have a robust way of validating each packet.

This is additional commentary about your situation:

In his comment, MartinJames is correct, though perhaps a bit blunt. Serial data streams without well-defined packet protocols are notoriously unreliable and data logging over such an interface is likely to produce erroneous data, which can have serious consequences if you’re doing research or engineering on the resulting dataset. A more robust message system might start each “packet” with a known character or character pair, just as a helpful resync mechanism: If your byte stream gets out-of-sync, the resync character (or pair) helps you get back in sync quickly & easily. In your case since you’re reading ASCII data, that's a '\n' or "\r\n", so from that standpoint you're good, as long as you actually do something to start & stop each data sample on those boundaries. What happens if you receive a data sample like this?...

01,03,CDCC1242,28,
01,03,CDCC1240,27,
01,03,CDCC1241,29,
01,03,CDCC1243,28,
01,03,CDCC123F,2A,
01,03,CD9,
01,03,CDCC1241,29,
01,03,CDCC1241,29,
01,0yĔñvśÄ“3,CDCC1243,28,
01,03,CDCC123F,2A,
01,03,CDCC1242,29,

Will your code be able to re-sync after the sample that’s missing several characters? What about the one that has garbage in it? Your code needs to be able to break apart the serial stream into chunks beginning with one delimiter character (or pair) and ending right before the next one in the serial stream. And it should examine the characters between and verify that they “make sense” in some manner, and be capable of rejecting any sample that doesn’t check out OK. What it does then may depend on the needs of your end consumer of the data: Perhaps you can just throw out the sample & still be OK. Or maybe you should repeat the last good sample until you get to the next good one. Or perhaps you should wait until the next good one and then linearly interpolate to find reasonable estimates of what the data should have been between those good samples.

In any case, as I said, you need some way to validate each data sample. If the “packet” (data sample) length can vary, then each packet should contain some indication as to the number of bytes in it, so if you get more or less, you know that packet is bad. (Also, if the length is unreasonable, you also know the data is bad, and you don’t allow your data collection algorithm to be fooled by a bad byte that your next packet is 1.8 Gigabytes long… which would probably crash your program since your receive buffer isn’t that big.) Finally, there should be some sort of checksum system over all the data in the packet; a 16-bit additive checksum would work, but a CRC would be better. By generating this packet overhead metadata on the sending end and verifying it on the receiving end, you (at least with some high probability) guarantee the validity of your dataset.

But as you said, you have no control over the format of the transmitted data. And that’s a shame; as MartinJames said, whoever designed the protocol didn’t seem to understand the unreliability of simple serial bytestreams. Since you can’t change that, you’ll just have to do your best to find some heuristics to validate the data; perhaps you make your code remember the last 5 samples in an array, and compare each new one to the last 5 assumed-valid samples; if you get a value that’s outside of the bounds of a reasonable change from the preceding samples, you throw it out & wait for the next one. Or come up with your own heuristics. Just make sure your heuristics don’t result in you invalidating all future samples if the actual measured value changes too fast.

Answer 2

You don't seem to have any message delimiters to indicate the start or end of a message stream. I assume that this is because you are working on only receiving ASCII data.

One option would look at would be to use strtok to split the data into strings (using ',' as the delimiter).

Test that you have 4 strings returned in your array.

Then for the first block just use atoi to convert into an integer. Doing this "001" and "01" should both convert to 1.

Ideally you should check the format of the message on reception, in case you haven't received a full message, but from what I can see here so far that isn't really necessary. Just check the format of each string e.g. if they contain non-numeric characters when the should then discard the data up to and including that point.

edit

I haven't understood how Temp is encoded but I have this example code Temp is incorrect in this code:

#include <stdio.h>

#include <stdlib.h>
#include "string.h"

int main()
{
    char input[] = "001,03,66661242,28";

    char* pstr = strtok(input,",");
    int count =0;
    int ID =0;
    int Mode =0;
    double Temp =0.0;
    float Volt = 0.0;

    while(pstr!=NULL)
    {
        switch(count)
        {
            case 0:
                ID = atoi(pstr);
            break;
            case 1:
                Mode = atoi(pstr);
            break;

            case 2:
                Temp = strtod(pstr, NULL);
            break;

            case 3 :
                Volt = strtol(pstr, NULL ,16)/10;
            break;

        }
        printf("%s\n", pstr);
        pstr = strtok(NULL,",");
        count++;

    }

    if(count == 4)
    {
        printf("ID = %d\n", ID);
        printf("Mode = %d\n", Mode);
        printf("Temp = %.1f\n", Temp);
        printf("Voltage = %.1f\n", Volt);
    }
    else
    {
        printf("Error");

    }

}