CODEWITHSUNDEEP
phpc++performancestliostream
BarsMonster
BarsMonster

Reputation: 6585

C++: Improving ifstream binary file reading speed

I am rewriting small program from PHP to C++. The idea is basically to read through 32Gb file on an SSD and do some simple operations on it.

I am using Visual Studio 2012 with x64 release build. PHP is 5.3 32bit.

The problem is that bare reading speed in PHP is higher, than in C++, and this really puzzles me. PHP does ~350 Mb/s and C++/ifstream code does 180 Mb/sec.

Code is really simple: 

ifstream datafile("data.txt", ios::binary);

while(datafile.read((char*)buffer, data_per_chunk)) {
//  do stuff;

I've tried different buffer sizes up to 16Mb and it did little difference. I also tried to set internal buffer via datafile.rdbuf()->pubsetbuf(...) but it also didn't made a difference.

Is there any hints on how to speed ifstream up without reverting to ancient C-level interface? I would like to at least reach PHP level of performance. Maybe some fancy read-ahead / cache settings or something.

I understand that memory-mapped files could likely help, but would prefer to tweak settings of ifstream, if it's possible to keep things simple given that file is significantly larger than physical RAM and larger than 4Gb i.e. no-go for potential 32-bit builds.

Upvotes: 2

Views: 3964

Answers (2)

BarsMonster
BarsMonster

Reputation: 6585

It appeared that you can reach maximum SSD reading speed even with ifstream.

To do so, you need to set internal ifstream readbuffer to ~2Mb, which is where peak SSD read speed happening, while fitting nicely in L2 cache of CPU. Then you need to readout data in chunks smaller than internal buffer. I've got best results reading data in 8-16kB chunks, but it only about 1% faster than reading in 1Mb chunks.

Setting ifstream internal buffer:

ifstream datafile("base.txt", ios::binary);
datafile.rdbuf()->pubsetbuf(iobuf, sizeof iobuf);

With all these tweaks I've got 495 Mb/sec read speed which is close to theoretical maximum of M500 480Gb SSD. During execution CPU load was 5%, which means that it was not really limited by ifstream implementation overhead.

I found no observable speed difference between ifstream and std::basic_filebuf.

Upvotes: 6

Kuba hasn't forgotten Monica
Kuba hasn't forgotten Monica

Reputation: 98425

I don't see the point of using ifstream when you're reading it all into a buffer. Either basic_filebuf or the "ancient" C interface will work. You need to compare ifstream to the C interface first, so that you know it's really ifstream to blame.

I see the following options, in order of increasing performance:

  • std::ifstream: read, etc.
  • std::basic_filebuf: open, sgetn, etc.
  • C: fopen, fread, etc.
  • WinApi: CreateFile (not OpenFile!), ReadFileEx, etc.

Perhaps PHP is not using the C interface internally, but winapi, and that is where the difference comes from.

Upvotes: 1

Related Questions