How to read from the terminal "keystrokes buffer"?

Question

Say that the ping command is running, and I type something on the terminal while ping is still running.

Now when ping terminates and bash gain back control, bash will print on the terminal what I typed while ping was running. This is a screenshot that shows what I mean:

How did bash get this information? I am sure it did not get it from stdin, because when I typed "I typed this while ping was running", I did not press Enter (and so stdin is empty).

So this data must have been stored in a "keystrokes buffer", and bash read from this buffer.

My question is, how does bash reads from this buffer (what function does it call...)?

Answer 1

It's a thing with the readline library (here's a more approachable page about it).

You can see this through Python, which is compiled with readline support on most distros:

>>> import time
>>> time.sleep(5)
I am typing this during the sleep>>> I am typing this during the sleep

However, I also happened to have one without readline support:

>>> import time
>>> time.sleep(5)
I am typing this during the sleep>>> 

(This can also be achieved through cat | python -i, because cat doesn't use readline, and python disables readline because its input is not a terminal.)

My guess as to what happens is:

1. Readline disables buffering (probably something like this). This way, it can receive all the characters as they're typed.
2. Readline disables echoing the characters to the terminal, and takes over control. (Here's a possible way.)
3. Bash disables readline so echo can do its stuff.
4. Echo ignores the characters, so they're still in stdin after being typed.
5. Readline helpfully takes back control, disables buffering/echoing typed characters (too late for the ones already typed, though), and handles the characters from stdin.

TL;DR: They're automatically echoed once, and then again by the readline library.

Answer 2

This is what I think happens:

1. When bash is running, it disables the tty device line buffering, it also disables the tty device echoing (now bash is the one that takes the responsibility of echoing everything it receives).
2. Immediately before bash executes ping, bash re-enables the tty device line buffering and the tty device echoing. Then bash executes ping.
3. Now I type"I typed this while ping was running" while ping is displaying its output. The tty device will echo back this string to the terminal window, but also this string is now buffered in the tty device.
4. ping terminates and bash gains back control.
5. bash re-disables the tty device line buffering and the tty device echoing. Disabling the tty device line buffering will cause the tty device line buffer to be flushed to bash, and bash will display it on the terminal window.

Answer 3

There is no special "read the terminal queue" mechanism going on in bash. It's just ordinary read and write system calls.

On Linux, the "tty device" is always buffered. If you type faster than the receiving program can read the input, the characters are not dropped into the bit bucket; they are placed in the input queue for the terminal device, from which they can be retrieved with a read system call.

The read call has the prototype ssize_t read(int fd, void *buf, size_t count);. When the terminal driver decides that the read call should return, it removes count bytes from the terminal's input queue and places them into buf. If the queue does not contain count bytes, then all of the bytes in the queue are read; if it contains more than count bytes, the remaining bytes remain in the queue for the next read.

count has no impact on the operation of the terminal itself; read(0, buf, 1) does not cause the read to return when one byte is available. It only limits the number of bytes placed into buf.

There are. however, a number of control settings which affect how read calls are handled. The most important one, in this context, is the ICANON flag. If this flag is set ("canonical mode"), a process waiting on a read system call on the tty will not be woken until an NL character is typed. (Actually, there are four characters which will wake the process up: NL, EOL, EOL2 and EOF.) In canonical mode, the kernel driver also handles some line editing characters, such as the ERASE character. (All of these characters are configurable through termios so that when I say "NL character", I mean "the character currently configured as NL. By default, the NL character is the "Enter" key and EOF is control-D.)

If ICANON is not set, the terminal is in non-canonical mode, and the VMIN and VTIME settings apply. VMIN is the minimum number of characters which must be present before the process is woken up; VTIME is the minimum amount of time which the terminal driver will wait for input before giving up and waking the process up. If both VMIN and VTIME are set, then the kernel driver will wait (indefinitely) for the first character, and will then wait up to VTIME for each successive character until VMIN characters are read. If neither VMIN nor VTIME is set, the read call will always return immediately.

Independent of the canonical mode setting, you can also configure the terminal's echoing behaviour. In the simplest configurations, you either set ECHO or not. If ECHO is set, printable characters are echoed by the terminal driver when they are typed. If ECHO is not set, the terminal driver does not echo characters. By default, ECHO is set.

Bash normally uses the readline library to handle terminal input. You can tell bash not to use readline, in which case the behaviour will be different; I'm only going to describe the normal case, with readline.

While bash is active and in control of the terminal, readline places the terminal into non-canonical mode and turns off echoing. It sets VMIN to 1 so read calls will return as soon as a single character is typed, and then goes into something like the following loop:

while (1) {
  ssize_t nread = read(0, buf, 1);
  if (nread && isprint(buf[0]))
    write(1, buf, 1);  /* Echo the character */
  else {
    /* Lots of complicated logic to handle other characters */
  }
}

Just before bash lets a child process execute, it resets the terminal by putting it back into canonical mode and turning echo on. Unless the command being executed changes the terminal mode, it stays like that until the command exits, at which point bash regains control, turns canonical mode and echoing off, and goes back into the readline loop.

So suppose that the command being executed does not change the terminal settings, and executes for some period of time (as in the ping example in your question). While ping is executing the terminal is in canonical mode, echoing is on, and no-one is reading from the terminal device. So anything you type will be placed into the terminal queue and echoed. "Anything" includes things which would normally terminate a read, like the Enter key. That's because there is no read to terminate.

When ping finally finishes, bash changes the terminal back to non-canonical mode and turns echoing off, so that characters you now type will not be echoed by the terminal driver. It then calls readline which starts up the loop above. However, at this point there is a bunch of stuff in the terminal queue, considerably more than the one character required by the VMIN setting. So the read call returns immediately with one character, and the write call in that loop then echoes the character which was just read. Of course, that character had already been echoed by the terminal driver, but that fact is not magically recorded in the character's binary encoding; it's just an ordinary character and so it gets echoed a second time. This continues until either the terminal queue is emptied or some character you previously typed requires readline's attention. If the terminal queue just contains some ordinary printable characters, they will all get echoed and the next read call will block until you type something.