Fastest output to file in c and c++

Question

I was helping someone with a question about outputting in C, and I was unable to answer this seemingly simple question I wanted to use the answer to (in my answer), that is:

What's the fastest way to output to a file in C / C++?

I've done a lot of work with prime number generation and mathematical algorithm optimization, using C++ and Java, and this was the biggest holdup for me sometimes - I sometimes need to move a lot to a file and fast.

Forgive me if this has been answered, but I've been looking on google and SO for some time to no avail.

I'm not expecting someone to do the work of benchmarking - but there are several ways to put to file and I doubt I know them all.

So to summarize,

What ways are there to output to a file in C and C++?

And which of these is/are the faster ones?

Obviously redirecting from the console is terrible. Any brief comparison of printf, cout, fputc, etc. would help.

Edit:

From the comments,

There's a great baseline test of cout and printf in: mixing cout and printf for faster output

This is a great start, but not the best answer to what I'm asking. For example, it doesn't handle std::ostreambuf_iterator<> mentioned in the comments, if that's a possibility. Nor does it handle fputc or mention console redirection (how bad in comparison)(not that it needs to)

Edit 2:

Also, for the sake of arguing my historical case, you can assume a near infinite amount of data being output (programs literally running for days on a newer Intel i7, producing gigabytes of text)

Temporary storage is only so helpful here - you can't buffer gigabytes of data easily that I'm aware.

Answer 1

The functions such as fwrite, fprintf, etc. Are in fact doing a write syscall. The only difference with write is that these functions use a buffer to reduce the number of syscalls.

So, if I need to choose between fwrite, fprintf and write, I would avoid fprintf because it's a nice but complicated function that does a lot of things. If I really need something fast, I would reimplement the formating part myself to the bare minimum required. And between fwrite and write, I would pick fwrite if I need to write a lot of small data, otherwise write could be faster because it doesn't require the whole buffering system.

Answer 2

The bottleneck in performance of output is formatting the characters.

In embedded systems, I improved performance by formatting text into a buffer (array of characters), then sending the entire buffer to output using block write commands, such as cout.write or fwrite. The functions bypass formatting and pass the data almost straight through.

You may encounter buffering by the OS along the way.

The bottleneck isn't due to the process of formatting the characters, but the multiple calls to the function.

If the text is constant, don't call the formatted output functions, write it direct:

static const char  Message[] = "Hello there\n";
cout.write(&Message[0], sizeof(Message) - 1);  // -1 because the '\0' doesn't need to be written

Answer 3

As far as I'm aware, the biggest bottleneck would be to write a character at a time (for example, using fputc). This is compared to building up a buffer in memory and dumping the whole lot (using fwrite). Experience has shown me that using fputc and writing individual characters is considerably slower.

This is probably because of hardware factors, rather than any one function being faster.

Answer 4

cout is actually slightly faster than printf because it is a template function, so the assembly is pre-compiled for the used type, although the difference in speed is negligible. I think that your real bottle neck isn't the call the language is making, but your hard-drives write rate. If you really want to go all the way with this, you could create a multi-thread or network solution that will store the data in a buffer, and then slowly write the data to the a hard-drive separate from the processing of the data.