Effective way of reducing data for real-time plot

Question

I am developing scientific application in Windows Forms (VC++ 2010), which controls relatively new, electronic device. I control it by additional, wrapped library written in C. After initial setup of all parameters, this application triggers a measurement in the device. Then, it sends to my app a huge data of over 200k samples of int at significant rate – let’s assume it’s 50 datasets per second.

Now, I need to plot my data at the real-time pace using Windows Forms chart. It would be perfect to have 750 samples plotted inside chart at rate of about 30 FPS. The problem I encountered lies in the algorithm of reducing database in a fast way without losing reliability of plot.

My ideas (data is oscilating around value = 127):

1. Choose 750 points just by selecting every (200 000/ 750) th point

2. Group the data and calculate mean value

3. Group the data and select maximum or minimum (based on overall group placement – if most of them is above 127 – select minimum, else maximum).

Which one (if any) of those solution is the best considering I have to plot data at real-time speed and plot should not miss spots, where we have any significant signal (looking like a kind of narrowed, modulated sine wave)? Is there any better approach?

And the last question: should I consider using table of pointers to my huge data buffer or data copies as data for plot considering I always have the same buffer of collected data (device just overwrites this buffer constantly with new data)?

This is my first post, so please inform me if there will be anything wrong in the style of post.

Answer 1

I developed an application that reads data at 256Hz (256 samples / second) from 16 channels and displays it in 16 different charts. The best way of plotting all data in real time was using a separate thread to updoat the plots. Here is the solution (in c#) that might be useful for you too.

When new data is read, data is stored in a list or array. Since it is real-time data, the timestamps are also generated here. Using the sample rate of the data acquired: timeStamp = timeStamp + sampleIdx/sampleRate;

public void OnDataRead(object source, EEGEventArgs e)
        {
            if ((e.rawData.Length > 0) && (!_shouldStop))
            {
                lock (_bufferRawData)
                {
                    for (int sampleIdx = 0; sampleIdx < e.rawData.Length; sampleIdx++)
                    {
                        // Append data
                        _bufferRawData.Add(e.rawData[sampleIdx]);

                       // Calculate corresponding timestamp
                      secondsToAdd = (float) sampleIdx/e.sampleRate;

                    // Append corresponding timestamp
                    _bufferXValues.Add( e.timeStamp.AddSeconds(secondsToAdd));
                    }
                }

Then, create a thread that sleeps every N ms (100ms is suitable for me for a 2 seconds display of data, but if I wanna display 10 seconds, I need to increase to 500ms of sleep time for the thread)

 //Create thread
 //define a thread to add values into chart
 ThreadStart addDataThreadObj = new ThreadStart(AddDataThreadLoop);
 _addDataRunner = new Thread(addDataThreadObj);
 addDataDel += new AddDataDelegate(AddData);

 //Start thread
 _addDataRunner.Start();

And finally, update the charts and make the thread sleep every N ms

 private void AddDataThreadLoop()
    {
        while (!_shouldStop)
        {
            chChannels[1].Invoke(addDataDel);

            // Sleeep thread for 100ms
            Thread.Sleep(100); 
        }
    }

Data will be added to the chart every 100ms

private void AddData()
    {
        // Copy data stored in lists to arrays
        float[] rawData;
        DateTime[] xValues;

            if (_bufferRawData.Count > 0)
            {
                // Copy buffered data in thread-safe manner
                lock (_bufferRawData)
                {
                    rawData = _bufferRawData.ToArray();
                    _bufferRawData.Clear();
                    xValues = _bufferXValues.ToArray();
                    _bufferXValues.Clear();
                }

                for (int sampleIdx = 0; sampleIdx < rawData.Length; sampleIdx++)
                {
                        foreach (Series ptSeries in chChannels[channelIdx].Series)
                            // Add new datapoint to the corresponding chart (x, y, chartIndex, seriesIndex)
                            AddNewPoint(xValues[sampleIdx], rawData[sampleIdx], ptSeries);
           }
      }
 }