CODEWITHSUNDEEP
c++iosolid-state-drive
digitalPhonix
digitalPhonix

Reputation: 170

Writing many large files quickly in C++

I have a program which gets a stream of raw data from different cameras and writes it to disk. The program runs these sorts of recordings for ~2 minutes and then another program is used to process the frames.

Each raw frame is 2MB and the frame rate is 30fps (ie. data rate is around 60MB/s) and I'm writing to an SSD which can easily handle a sustained > 150MB/s (tested by copying 4000 2MB files from another disk which took 38 seconds and Process Explorer shows constant IO activity).

My issue is that occasionally calls to fopen(), fwrite() and fclose() stall for up to 5 seconds which means that 300MB of frames build up in memory as a back log, and after a few of these delays I hit the 4GB limit of a 32 bit process. (When the delay happens, Process Explorer shows a gap in IO activity)

There is a thread which runs a loop calling this function for every new frame which gets added to a queue:

writeFrame(char* data, size_t dataSize, char* filepath)
{
    // Time block 2
    FILE* pFile = NULL;
    fopen(&pFile, filepath, "wb");
    // End Time block 2

    // Time block 3
    fwrite(data,1,dataSize,pFile);
    // End Time block 3

    // Time block 4
    fclose(pFile);
    // End Time block 4
}

(There's error checking too in the actual code but it makes no difference to this issue) I'm logging the time it takes for each of the blocks and the total time it takes to run the function and I get results which most of the time look like this: (times in ms)

TotalT,5,       FOpenT,1,       FWriteT,2,    FCloseT,2
TotalT,4,       FOpenT,1,       FWriteT,1,    FCloseT,2
TotalT,5,       FOpenT,1,       FWriteT,2,    FCloseT,2

ie. ~5ms to run the whole functions, ~1ms to open the file, ~2ms to call write and ~2ms to close the file.

Occasionally however (on average about 1 in every 50 frames, but sometimes it can be thousands of frames between this problem occurring), I get frames which take over 4000ms:

TotalT,4032,    FOpenT,4023,    FWriteT,6,    FCloseT,3

and

TotalT,1533,    FOpenT,1,       FWriteT,2,    FCloseT,1530

All the frames are the same size and its never fwrite that takes the extra time, always fopen or fclose

No other process is reading/writing to/from this SSD (confirmed with Process Monitor).

Does anyone know what could be causing this issue and/or any way of avoiding/mitigating this problem?

Upvotes: 0

Views: 724

Answers (2)

CoffeDeveloper
CoffeDeveloper

Reputation: 8317

Prepare empty files (2 MB files filled with zeros) So that space is already "ready", then just overwrite these files. Or create a file that is a batch of several frames, so you can reduce number of files.

there are libraries for doing compression and decompression and playback of videos:

libTheora may be usefull because already compress frames (well you will need to output the video in a single file) and do that pretty fast (lossy compression by the way).

Upvotes: 1

MSalters
MSalters

Reputation: 179779

I'm going to side with X.J., you're probably writing too many files to a single directory. A solution could be to create a new directory for each batch of frames. Also consider calling SetEndOfFile directly after creating the file, as that will help Windows allocate sufficient space in a single operation.

FAT isn't a real solution as it's doing even worse on large directories.

Upvotes: 1

Related Questions