multi pipes in C

Question

I am trying to implement multiple pipes in C, the solution should be both for:

cmd1 | cmd2 | cmd3

and for:

        |--- cmd2

cmd1    |--- cmd3

        |--- cmd4

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <signal.h>

int main(int argc, char *argv[]) {

char* args1[] = { "ls", NULL, NULL };
char* args2[] = { "ls", "-l", NULL };
char* args3[] = { "sort", NULL, NULL };
char* args4[] = { "wc", "-l", NULL };

int rc1 = execute_cmd(args1, 0);
//printf("rc1 = %d\n", rc1);

int rc2 = execute_cmd(args2, rc1);
//printf("rc2 = %d\n", rc2);

int rc3 = execute_cmd(args3, rc1);
//printf("rc3 = %d\n", rc3);

int rc4 = execute_cmd(args4, rc1);
//printf("rc4 = %d\n", rc4);

int buffer[1024];
int len = 0;

if (rc2) {
    while ((len = read(rc2, buffer, sizeof(buffer))) > 0) {
        write(STDERR_FILENO, "rc2\n", 4);
        write(STDERR_FILENO, &buffer, len);
    }
} else {
    printf(stderr, "ERROR\n");
}

if (rc3) {
    while ((len = read(rc3, buffer, sizeof(buffer))) > 0) {
        write(STDERR_FILENO, "rc3\n", 4);
        write(STDERR_FILENO, &buffer, len);
    }
} else {
    printf(stderr, "ERROR\n");
}

if (rc4) {
    while ((len = read(rc4, buffer, sizeof(buffer))) > 0) {
        write(STDERR_FILENO, "rc4\n", 4);
        write(STDERR_FILENO, &buffer, len);
    }
} else {
    printf(stderr, "ERROR\n");
}

return 0;
}

int execute_cmd(char** args, int fd_in) {

int pipefd[2];
pipe(pipefd);

if (fork() == 0) {
    close(pipefd[0]);

    dup2(pipefd[1], STDOUT_FILENO);
    dup2(pipefd[1], STDERR_FILENO);

    close(pipefd[1]);

    if (fd_in) {
        dup2(fd_in, 0);
    }

    execvp(*args, args);
    printf("failed to execute %s %s", *args, *args[0]);
} else {
    close(pipefd[1]);

    return pipefd[0];

}
}

The output of the program is not deterministic, once I see the right result, and once I saw different result. Looks like dup2 not function as I expected, if I dup2 multiple times and for each file descriptor read from the result file descriptor - it's look like it's influence on the copied file descriptor?

If it's work like I mention by design, which system call I need to use for both?

Answer 1

Having multiple handles/references to the same pipe will cause a lot of problems with synchronisation, etc.

For example, if there's 2 child processes where one sends " Hello\n" then " World\n" and the other sends " Foo\n" then " Bar\n"; then you could end up with " Hello\n World\n Foo\n Bar\n" or " Hello\n Foo\n World\n Bar" or " Foo\n Hello\n Bar\n World", etc. The output ends up unordered (which would be extremely confusing).

The solution is to use different pipes.

Basically, when the main program forks it should create new pipe/s that will become the child process' STDOUT and STDERR. Then the main program will need to read from its end of all the new pipes and (potentially) buffer the information, so that the main process can send the data from children to its own STDOUT/STDERR in a specific order - e.g. all of the first child's output, then all of the second child's output, then all of the next child's output, etc.

The main program could also add extra information and do some formatting to make it clearer what is happening. For the example above, you could end up with:

Process A (exit status = 0, OK):
    Hello
    World
Process B (exit status = 1, Failed):
    Foo
    Bar

Rather than just:

 Hello
 World
 Foo
 Bar

For input (STDIN) I've got no idea how you want it to work. If none of the child processes need STDIN (the easiest and most likely case) then you can ignore it. If each child process needs to get their own copy of the main process' STDIN then you'd need to create new pipes for each child to use as its STDIN.

The other alternative is to have a "currently selected child", which can become a lot more complex (especially if there's an end-user who needs to be able to see the output of the selected child, as that'd mean implementing some way of switching between "currently being displayed" children - e.g. when the user selects a different child, clear the screen and display that child's backlog).

Answer 2

Yes, dup and dup2 create completely equivalent handles to the same pipe. If several processes (or threads) simultaneously attempt to read from the pipe using duplicated/forked descriptors, a "random" of them will get to the data first, but each byte written to the pipe only gets delivered once.

If you want to copy data to multiple different readers, you have to program that explicitly -- fork a subprocess (or spawn a thread) to read some data from one incoming pipe, then write it to all of the outgoing ones, and continue in a loop until you reach EOF.