Stream vs Buffer

Question

there I'm new to C. I'm currently reading the K&R. There I got confused by a definition in it about the text streams "A text stream is a sequence of characters divided into new lines;each line consists of 0 or more characters followed by a newline character." And trying to knowing about this streams I was introduced to a new term namely buffer.

I just know that:

• A continuous flow of data (bytes or characters) in between Input and Output devices is
  a STREAM .
• A temporary storage area in main memory to store the input or output data temporarily is a BUFFER.

I don't say that I'm right, but it's my basic idea upon those terms.

I want to know, what actually buffer & stream are and how these 2 things(i.e, stream & buffer) work together, in the non-abstract level of C implementation.

Answer 1

You have three streams in C, stdin, stdout, and stderr, you can also think of files you have opened with fopen for example as a stream. stdin is generally the keyboard, stdout is generally your monitor, stderr is generally also your monitor. But they don't have to be, they are abstractions for the hardware.

If for example you didn't have a keyboard but a keypad on a bank ATM for example, then stdin would be the keypad, if you didn't have a monitor but instead had a printer, then stdout would be the printer. You change what hardware they are using with calls to your operating system. You can also change their behaviour, again, through calls to your operating system, which is beyond the scope of what you're asking.

So in a way, think of the buffer as the memory allocated by the operating system associated with the stream to hold the data received from the hardware component. When you type at your keyboard for example the characters you type aren't being capture directly by your IDE, they are moving from there into the buffer, then you read the buffer.

That's why, for example, you have to hit enter before your code starts interacting with whatever you typed into the keyboard, because stdin is line buffered. Control passes from your program to the operating system until it encounters something that sends control back to your program, in a normal situation that would be the newline character.

So in a way, think of it like this, the stream is the device (keyboard, monitor, or a file on your hard drive), the buffer is where the data is held while the operating system has control, and then you interact with the buffer while you are processing the data.

That abstraction allows you to use all of these different things in a common manner regardless of what they are, for example: fgets(str, sizeof(str), STREAM) ... stream can be any input stream, be it stdin or a file.

Taking it a step further that's why new programmers get thrown off by scanf for an int followed by an fgets, because scanf reads the int from the buffer but leaves the \n in the buffer ... then the call to fgets reads the \n that scanf left there and the new programmer is left wondering why they were unable to input any data. So your curiosity about streams and buffers will serve you well as you move forward in your learning about C.

Answer 2

Those are actually pretty good working definitions.

In practical C terms, a buffer is an array (usually of char or unsigned char type) that's used to store data, either as a result of an input operation, or before sending to output. The array can be declared as a fixed size array, such as

char buffer[SOME_BUFFER_SIZE];

or dynamically, using

char *buffer = malloc( SOME_BUFFER_SIZE * sizeof *buffer );

The advantage of using dynamic memory is that the buffer can be resized if necessary; the disadvantage is you have to manage the lifetime of that memory.

For text input/output you'd typically use arrays of char; for binary input/output you'd typically use arrays of unsigned char.

It's fairly common for systems communicating over a network to send data in fixed-size "chunks", such that you may need several read or write operations to get all the data across. Think of a Web server and a browser - the server sends the HTML in multiple messages, and the browser stores the intermediate result in a buffer. It's only when all the data has been received that the browser renders the page:

Received from Web server       Stored in browser's input buffer
------------------------       --------------------------------
HTTP/1.1 200 OK \r\n           <!DOCTYPE HTML><html
Content-length: 20\r\n
<!DOCTYPE HTML><html

HTTP/1.1 200 OK \r\n           <!DOCTYPE HTML><html><head><title>This i
Content-length: 20\r\n
><head><title>This i

HTTP/1.1 200 OK \r\n           <!DOCTYPE HTML><html><head><title>This i
Content-length: 20\r\n         s a test</title></he
s a test</title></he

HTTP/1.1 200 OK \r\n           <!DOCTYPE HTML><html><head><title>This i
Content-length: 20\r\n         s a test</title></head><body><p>Hello, W
ad><body><p>Hello, W

HTTP/1.1 200 OK \r\n           <!DOCTYPE HTML><html><head><title>This i
Content-length: 19             s a test</title></head><body><p>Hello, W
orld!</body></html>            orld!</body></html>

No sane server sends HTML in chunks of 20 characters, but this should illustrate why and how buffers get used.

Answer 3

Two common abstractions of I/O devices are:

Streams - transfers a variable number of bytes as the device becomes ready.

Block - transfers fixed-size records.

A buffer is just an area of memory which holds the data being transferred.

Answer 4

The deifinitions are not bad, actually very good. You could add (from an object oriented perspective), that a STREAM uses a BUFFER.

The use of a BUFFER might be necessary, e.g. performance reasons, since every system call comes with a relatively high cost.

Especially IO system calls, Harddisk or Network access are slow, compared to memory access times. And they add up if a read or write consists only of a single byte.