How piping mechanism works in UNIX

Question

I would like to confirm my understanding about piping in UNIX command execution.

When we use pipe in a sequence of commands, the output of preceding command is passed as input to the following command.

As we know, everything in UNIX is a file. And also every process (a.k.a: command) has its own dedicated files for STDIN and STDOUT. We can recall that these are not directly linked to terminal.

Let us consider the below example:

Assume that abc.txt is a file present in the working directory.

cat abc.txt | cat

Above, the first command cat abc.txt writes the output (content of the file abc.txt) to its dedicated standard output file (/proc/<processID>/fd/1 otherwise STDOUT ). The next cat command supposes to read from its own standard input file (/proc/<processID>/fd/0 otherwise STDIN ).

The STDOUT file in first cat command is completely different from the STDIN file of the second cat command. Both the cat commands are different processes and hence the would be different. And hence the STDIN and STDOUT files.

My understanding is:

The UNIX operating system (when it encounters piping mechanism), either copies the contents of the STDOUT file of the preceding command to the STDIN file of the following command OR makes symbolic reference of the previous command's STDOUT file to the following command's STDIN file.

Am I correct here?

Answer 1

You are absolutely correct! For those that are curious about the gory details... (Corrected as per comments)

Inside each process is an array of pointers to file structs stored in the kernels memory. These file structs are allocated and deallocated in kernel memory when your program ultimately calls the file open, close, etc system calls, and a pointer to them is stored in the process's file array. The index of that array stored in the process is nothing other than the file descriptor (fd). So when you pass a system call a file descriptor, the operating system is able to look up what file that references for the process in question, or set it to whatever was passed in.

Typically index 0, 1 and 2 are reserved by convention, and we call those STDIN, STDOUT, and STDERR, but remember they are just indexes into an array that contain pointers to an open "file". It is also important to note that the pipe system call sets aside two file descriptors, one for the "read" end of the file, and one for the "write" end of the file. The actual file in question isn't something on disk, instead it is a chunk of memory set aside in the kernel (when you call the pipe system call) that can be written to using the write system call on the "write" file descriptor and read from using the read system call on the "read" file descriptor.

Now the magic of what allows your shell program to run chained commands like that involves a complicated arrangement of forking, exec'ing, file closing, and the pipe system call. Ultimately though what ends up happening is that each process in the chain has its 1 file descriptor (STDOUT) set to the write end of the pipe, and has the 0 file descriptor (STDIN) set to the read end of the pipe. This creates a chain between the two processes such that they can read and write from a shared space in memory. The starting command and ending command in the chain end up defining where the initial data is coming from and where the eventual data is being written to.

To use your example above, say we have two processes, process A is the program running cat abc.txt, and process B is the program running cat. Your shell program will set up the processes such that:

Process A: fd 0 defaults to STDIN, fd 1 is the write end of the shared pipe in memory. Process B: fd 0 is the read end of the shared pipe in memory, and fd 1 is set to the terminal (which is usually because this is what STDOUT defaults to when it isn't overwritten).

As Process A runs the cat program it will open the file abc.txt, which will be placed into the first available fd (likely fd 3), and begin reading from fd 3 into fd 1. The neat thing about overwriting STDIN and STDOUT this way is that it allows the programs to be chained together, without knowing any details about what the other programs are in the chain. Setting up the file descriptors this way means your first cat will read in the abc.txt file, and write it to the shared pipe memory, and the second cat will read in the contents of the file from that shared pipe memory, writing it to the terminal.

My original description had Process A (cat abc.txt) reading the file from fd 0 STDIN. As the comments point out this would actually be the command cat < abc.txt | cat.

Things get a little bit more complicated since the reading and writing typically happens in chunks, which is common when using file read and file write system calls, but that doesn't really matter for the question. If you want an even more in-depth explanation, this blog post has some pretty pictures: https://www.rozmichelle.com/pipes-forks-dups/