CODEWITHSUNDEEP
clinuxglibc
Petr Skocik
Petr Skocik

Reputation: 60107

Reading directly from a FILE buffer

The core of my app looks approximately as follows:

size_t bufsize;
char*  buf1;
size_t r1; 
FILE* f1=fopen("/path/to/file","rb");
...
do{
  r1=fread(buf1, 1, bufsize, f1);
  processChunk(buf1,r1);
} while (!feof(f1));
...

(In reality, I have multiple FILE*'s and multiple bufN's.) Now, I hear that FILE is quite ready to manage a buffer (referred to as a "stream buffer") all by itself, and this behavior appears to be quite tweakable: https://www.gnu.org/software/libc/manual/html_mono/libc.html#Controlling-Buffering .

How can I refactor the above piece of code to ditch the buf1 buffer and use f1's internal stream buffer instead (while setting it to bufsize)?

Upvotes: 0

Views: 187

Answers (2)

Petr Skocik
Petr Skocik

Reputation: 60107

So I've read a little bit of the C standard and run some benchmarks and here are my findings:

1) Doing it as in the above example does involve unnecessary in-memory copying, which increases the user time of simple cmp program based on the above example about twice. Nevertheless user-time is insignificant for most IO-heavy programs, unless the source of the file is extremely fast. On in-memory file-sources (/dev/shm on Linux), however, turning off FILE buffering (setvbuf(f1, NULL, _IONBF, 0);) does yield a nice and consistent speed increase of about 10–15% on my machine when using buffsizes close to BUFSIZ (again, measured on the IO-heavy cmp utility based on the above snippet, which I've already mentioned, which I've tested on 2 identical 700MB files 100 times).

2) Whereas there is an API for setting the FILE buffer, I haven't found any standardized API for reading it, so I'm going to stick with the true and tested way of doing, but with the FILE buffer off (setvbuf(f1, NULL, _IONBF, 0);)

(But I guess I could solve my question by setting my own buffer as the FILE stream buffer with the _IONBF mode option (=turn off buffering), and then I could just access it via some unstandardized pointer in the FILE struct.)

Upvotes: 0

unwind
unwind

Reputation: 399979

If you don't want opaquely buffered I/O, don't use FILE *. Use lower-level APIs that let you manage all the application-side buffering yourself, such as plain POSIX open() and read() for instance.

Upvotes: 1

Related Questions