bLAZ
bLAZ

Reputation: 1751

Need for a fast writing of binary data to a file

I receive a fair amount of binary data from an external device, about 30-40 MB/s. I need to save it to a file. On the external source device side, I have a very small buffer that I can't enlarge and as soon as the transmission stutters on the C# application side, it quickly gets clogged and I lose data.

In the application, I tried writing using FileStream, but unfortunately it is not fast enough.

_FileStream = new FileStream(FileName, FileMode.Create, FileAccess.Write);

...

void Handler_OnFtdiBytesReceived(object sender, FtdiBytesReceivedEventArgs e)
{

    ...

    Array.Copy(e.Bytes, 0, _ReceivedDataBuffer, _ReceivedDataBufferPosition, e.NumBytesAvailable);

    _ReceivedDataBufferPosition += (int)e.NumBytesAvailable;

    if (_ReceivedDataBufferPosition > 0)
    {
        _FileStream.Write(_ReceivedDataBuffer, 0, (int)e.NumBytesAvailable);
        _ReceivedDataBufferPosition = 0;
    }

    if (_IsOperationFinished == true)
    {
        _FileStream.Flush();
        _FileStream.Close();
    }

    ...
}

I also tried adding a BinaryWriter:

_FileStream = new FileStream(FileName, FileMode.Create, FileAccess.Write);
_BinaryWriter = new BinaryWriter(_FileStream);

and then:

_BinaryWriter.Write(_ReceivedDataBuffer, 0, (int)e.NumBytesAvailable);

instead of previous _FileStream.Write(...), but also the buffer on the transmit side gets clogged.

Is there any way to deal with this? I wonder if it might help to somehow buffer the data in the computer's RAM when receiving it and, for example, when it reaches some sizable amount (say, 512 MB), start writing to the file in a separate Task, so that in the meantime, new data can be collected into the buffer continuously. Perhaps I would need to use two buffers and use them alternately, one to receive data continuously and the other from which to write to the file, and swap?? This seems quite complex code for someone with little experience, which I don't know if it will help, so I'd like to ask you for a hint first.

I'll just add at the end that I have the ability to watch the buffer fill up in this external device and by commenting out this one line regarding writing: _FileStream.Write(_ReceivedDataBuffer, 0, (int)e.NumBytesAvailable); I see that the problem with its clogging disappears. Earlier I also analyzed whether other code fragments might be inefficient, such as Array.Copy(...) or passing parameters via Event, but it had no effect.

Upvotes: 2

Views: 502

Answers (0)

Related Questions