CODEWITHSUNDEEP
c++file
TonyJ
TonyJ

Reputation: 71

SSD read for a large file using c++

I am working on a windows 10 64-bit machine, 6850K CPU, and 64 GB DDR4 RAM, with a Samsung SSD connected via M.2. I want to read a file that is about 15 GB in size, to memory. I am currently using fstream to read the entire file in to an array of unsigned chars using a single call to its read function. However, the speeds I achieve are not hitting the maximum read speeds of the SSD (1500 MB/s when the SSD read is around 3500 MB/s).

So I was wondering if there was a faster way? Would it be faster if I made multiple read calls for smaller chunks? If so, what is the optimal chunk size? I have seen some people mention 4K reads in some previously asked questions. Does that apply in this case?

Any help is appreciated.

My code excerpt is as follows

my read code is as follows

fstream myFile;
myFile.open("file", ios::binary | ios::in);
myFile.read(reinterpret_cast<char*>(buf), 14929920000LL);

where buf is the same size as the read.

Upvotes: 5

Views: 1407

Answers (1)

1201ProgramAlarm
1201ProgramAlarm

Reputation: 32732

To get the fastest read speed, you need to bypass the windows disk cache. Use Windows API calls CreateFile, ReadFile, etc. and use unbuffered reads (pass FILE_FLAG_NO_BUFFERING to CreateFile). This will transfer data directly from the disk to the memory block you need without having to copy the data from one memory address to another.

You'll need to pay close attention to the required memory alignment necessities that may be imposed by hardware. This typically requires memory addresses to be 512 byte aligned, but some newer hardware may want 4096 bytes, and others may not be as strict. A link in the CreateFile documentation gives the full details for FILE_FLAG_NO_BUFFERING.

Upvotes: 2

Related Questions