CODEWITHSUNDEEP
c++file-io
user1838343
user1838343

Reputation: 449

Improve performance of ofstream

I am writing strings to a text file using:

ofstream ofs("ofs.txt", ios_base::binary);
std::string str = "Hi";
for (int i = 0 ; i < 10000000 ; i++) {
    ofs << str.c_str();
    ofs << "\n" ;
}

However, this takes long time to execute. Can anybody help me how to improve performance of the above. Or any other faster way to writing strings to file.

Thanks.

Upvotes: 2

Views: 9467

Answers (3)

Mr.C64
Mr.C64

Reputation: 43044

In several cases I found that C++ I/O streams tend to be slower than C <stdio.h> FILE*.

I had a confirmation also in the following test:

#define _CRT_SECURE_NO_WARNINGS // for stupid fopen_s warning
#include <stdio.h>
#include <exception>
#include <fstream>
#include <iostream>
#include <ostream>
#include <stdexcept>
#include <string>
#include <vector>
#include <windows.h>
using namespace std;

long long Counter() 
{
  LARGE_INTEGER li;
  QueryPerformanceCounter(&li);
  return li.QuadPart;
}

long long Frequency() 
{
  LARGE_INTEGER li;
  QueryPerformanceFrequency(&li);
  return li.QuadPart;
}

void PrintTime(long long start, long long finish, const char * s) 
{
  cout << s << ": " << (finish - start) * 1000.0 / Frequency() << " ms" << endl;
}

// RAII wrapper to FILE*
class File
{
public:
  explicit File(FILE * f)
    : m_file(f)
  {}

  ~File()
  {
    fclose(m_file);
  }

  FILE* Get() const
  {
    return m_file;
  }

  bool IsOpen() const
  {
    return (m_file != nullptr);
  }

private:
  FILE* m_file;

  File(const File&);
  File& operator=(const File&);
};

void TestIoStream(const vector<string>& lines)
{
  ofstream ofs("ofs.txt", ios_base::binary);
  for(auto it = lines.begin(); it != lines.end(); ++it)
  {
    ofs << it->c_str();
    ofs << "\n" ;
  }
}

void TestStdioFile(const vector<string>& lines)
{
  File file( fopen("cfile.txt", "wt") );
  if (! file.IsOpen())
    throw runtime_error("Can't open C FILE*.");

  for(auto it = lines.begin(); it != lines.end(); ++it)
  {
    fputs( it->c_str(), file.Get());
    fputs( "\n", file.Get());
  }
}

int main()
{
  static const int kExitOk = 0;
  static const int kExitError = 1;
  try
  {
    cout << "Building test lines...";
    vector<string> lines;
    for (int i = 0; i < 10000000; i++)
      lines.push_back("Hi");
    cout << "done. ";
    cout << "(Count = " << lines.size() << ")" << endl;

    long long start = 0;
    long long finish = 0;

    start = Counter();  
    TestIoStream(lines);
    finish = Counter();
    PrintTime(start, finish, "C++ I/O stream");

    start = Counter();
    TestStdioFile(lines);
    finish = Counter();
    PrintTime(start, finish, "C FILE*");

    return kExitOk;
  }
  catch(const exception& e)
  {
    cerr << "\n*** ERROR: " << e.what() << endl;
    return kExitError;
  }
}

Compiled with VS2010 SP1 (VC10):

cl /EHsc /W4 /nologo /MT /O2 /GL test.cpp

Test result:

Building test lines...done. (Count = 10000000)
C++ I/O stream: 2892.39 ms
C FILE*: 2131.09 ms

Upvotes: 1

Mats Petersson
Mats Petersson

Reputation: 129524

The fastest way to write a lot of strings to a file is (most likely) via the C FILE * interface. It is certainly fairly well documented that the C++ stream interface isn't as performant as the FILE *.

Ideally, if you build up a buffer of a few KB (perfectly, 4KB at a time), and write that using fwrite(buffer, size_of_buffer, 1, outfile);, you should get pretty close to the performance the system can produce. Yes, you can probably tune it half a percent or so more by using the native interfaces of write or WriteFile for Linux/Unix and Windows respectively.

The drawback of course is that you don't have an easy way to simply add an output method for arbitrary new types.

As always, when it comes to performance, measure the different options, and see which comes out best. If it's something that is going to be used on many machines under different circumstances, it's also good to test with different hardware, different amounts of memory and system load, to determine that you are not making just your machine run faster, but that it actually gives reasonable improvement [or at least no bad drawback] on systems with different conditions.

Upvotes: 1

James Kanze
James Kanze

Reputation: 154047

Just using ofs << str; might be slightly faster (or it might not be, but it's certainly more idiomatic). But you're writing 20 megabytes of data; that's going to take some time, regardless of how you do it.

You don't say exactly what you're trying to do. (I'm guessing that the real problem doesn't involve ten million times "Hi".) Depending on that, it might be possible to use low level IO, or even mmap. But neither of these do any formatting, and if you have to format to a ostringstream first, you might find that they aren't significantly faster.

Upvotes: 1

Related Questions