How to use write() or fwrite() for writing data to terminal (stdout)?

Question

I am trying to speed up my C program to spit out data faster.
Currently I am using printf() to give some data to the outside world. It is a continuous stream of data, therefore I am unable to use return(data).

How can I use write() or fwrite() to give the data out to the console instead of file?

Overall my setup consist of program written in C and its output goes to the python script, where the data is processed further. I form a pipe:

./program_in_c | script_in_python

This gives additional benefit on Raspberry Pi by using more of processor's cores.

Answer 1

Well, there's little or no win in trying to overcome the already used buffering system of the stdio.h package. If you try to use fwrite() with larger buffers, you'll probably win no more time, and use more memory than is necessary, as stdio.h selects the best buffer size appropiate to the filesystem where the data is to be written.

A simple program like the following will show that speed is of no concern, as stdio is already buffering output.

#include <stdio.h>
int
main()
{
    int c;

    while((c = getchar()) >= 0)
        putchar(c);
}

If you try the above and below programs:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int
main()
{
    char buffer[512];
    int n;

    while((n = read(0, buffer, sizeof buffer)) > 0)
        write(1, buffer, n);

    if (n < 0) {
        perror("read");
        return EXIT_FAILURE;
    }

    return EXIT_SUCCESS;
}

You will see that there's no significative difference or, even, the first program will be faster, despite it is doing I/O on a per character basis. (as B. Kernighan & Dennis Ritchie wrote it in her first edition of "The C programming language") Most probably the first program will win.

The calls to read() and write() involve a system call each, with a buffer size decided by you. The individual getchar() and putchar() calls don't. They just store the received chars in a memory buffer, as you print them, whose size has been decided by the stdio.h library implementation, based on the filesystem, and it flushes the buffer, once it is full of data. If you grow the buffer size in the second program, you'll see that you get better results increasing it up to a point, but after that you'll see no more increment in speed. The number of calls made to the library is insignificant with respect to the time involved in doing the actual I/O, and selecting a very large buffer, will eat much memory from your system (and a Raspberry Pi memory is limited in this sense, to 1Gb or ram) If you end making swap due to a so large buffer, you'll lose the battle completely.

Most filesystems have a preferred buffer size, because the kernel does write ahead (the kernel reads more than what you asked for, on sequential reads, in prevision that you'll continue reading more after you consumed the data) and this affects the optimum buffer size. For that, the stat(2) system call tells you what is the optimum buffer size, and stdio uses that when it selects the actual buffer size.

Don't think you are going to get better (or much better) than the program listed first above. Even if you use large enough buffers.

What is not correct (or valid) is to intermix calls that do buffering (like all the stdio package's) with basic system calls (like read(2) or write(2) ---as I've seen recommending you to use fflush(3) after write(2), which is totally incoherent--- that do not buffer the data) there's no earn (and probably you'll get your output incorrectly ordered, if you do part of the calls using printf(3) and part using write(2) (this happens more in pipelines like you plan to do, because the buffers are not line oriented ---another characteristic of buffered output in stdio---)

Finally, I recomend you to read "The Unix programming environment" by Dennis Ritchie and Rob Pike. It will teach you a lot of unix, but one very good thing is that it will teach you to use perfectly the stdio package and the unix filesystem calls for reading and writing. With a little of luck you'll find it in .pdf on internet.

The next program shows you the effect of buffering:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int
main()
{
    int i;
    char *sep = "";

    for (i = 0; i < 10; i++) {
        printf("%s%d", sep, i);
        sep = ", ";
        sleep(1);
    }
    printf("\n");
}

One would assume you are going to see (on the terminal) the program, writing the numbers 0 to 9, separated by , and paced on one second intervals.

But due to the buffering, what you observe is quite different, you'll see how your program waits for 10 seconds without writing anything at all on the terminal, and at the end, writes everything in one shot, including the final line end, when the program terminates, and the shell shows you the prompt again.

If you change the program to this:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int
main()
{
    int i;
    char *sep = "";

    for (i = 0; i < 10; i++) {
        printf("%s%d", sep, i);
        fflush(stdout);
        sep = ", ";
        sleep(1);
    }
    printf("\n");
}

You'll see the expected output, because you have told stdio to flush the buffer at each loop pass. In both programs you did 10 calls to printf(3), but there was only one write(2) at the end to write the full buffer. In the second version you forced stdio to do one such write(2) after each printf, and that showed the data out as the program passed through the loop.

Be careful, because another characteristic of stdio can be confounding you, as printf(3), when you print to a terminal device, flushes the output at each \n, but when you run it through a pipe, it does it only when the buffer fills up completely. This saves system calls (in FreeBSD, for example, the buffer size selected by stdio is around 32kb, large enough to force two blocks to write(2) and optimum (you'll not get better going above that size)

Answer 2

#include <unistd.h>

       ssize_t write(int fd, const void *buf, size_t count);

write() writes up to count bytes from the buffer starting at buf to the file referred to by the file descriptor fd.

the standard output file descriptor is: 1 in linux at least! concern using flush the stdoutput buffer as well, before calling to write system call to ensure that all previous garabge was cleaned

fflush(stdout); // Will now print everything in the stdout buffer
write(1, buf, count);

using fwrite:

size_t fwrite(const void *ptr, size_t size, size_t nmemb, FILE *stream);

The function fwrite() writes nmemb items of data, each size bytes long, to the stream pointed to by stream, obtaining them from the location given by ptr.

fflush(stdout);
int buf[8];
fwrite(buf, sizeof(int), sizeof(buf), stdout);

Please refare to man pages for further reading, in the links below:

fwrite

write

Answer 3

The console output in C works almost the same way as a file. Once you have included stdio.h, you can write on the console output, named stdout (for "standard output"). In the end, the following statement:

printf("hello world!\n");

is the same as:

char str[] = "hello world\n";

fwrite(str, sizeof(char), sizeof(str) - 1, stdout);
fflush(stdout);