select() does not work for multiple named pipes

Question

I'm currently trying to implement a chat server/shoutbox via named pipes. The server takes the name of the pipes as a command line argument. It then proceeds with creating the pipes and opens them for reading. The other processes open the pipe for writing and ask the user for input (client). Once the user hits enter, the message is written to the pipe. When the user hits enter without entering any character the client closes the pipe and exits. The server does not need to support reconnects and the clients need to connect in the order they have been specified on the command line. This works fine for exactly one client, but not for a second or third one. My code to implement this behaviour is the following:

Side note: the hashtag plus client name serves to identify the client. It is sent in the beginning to register a client on the server, so that it knows to which client it is talking. In the file fifo.h respectively fifo.c there are my custom implementations of create_fifo, open_fifo and close_fifo which handle the creation, opening and closing of the fifos.

Edit: I'm a step closer now to what I have to achieve. I had to add a few lines to the code which I will mark in the code below by adding a comment. The only problem that persists is that the server does not recognize when the second user leaves the server first and then the first one does. So it leaves the pipes open and does not return to the clean_up function.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/fcntl.h>
#include <string.h>
#include "fifo.h"

void clean_up(int* fifos, char** argv, int n);

int main(int argc, char** argv){
    int clients = argc - 1;

    int fifos[clients];
    char reg[clients][PIPE_BUF];

    fd_set active_clients;
    FD_ZERO(&active_clients);

    for (int i = 0; i < clients; i++){
        create_fifo(argv[i + 1]);
        fifos[i] = open_fifo(argv[i + 1], O_RDONLY);

        FD_SET(fifos[i], &active_clients);
    }
    fd_set tmpset;
    memcpy(&tmpset, &active_clients, sizeof(fd_set)); // <-- I added this line

    setbuf(stdout, NULL);

    while (clients > 0){
        if (select(fifos[clients - 1] + 1, &active_clients, NULL, NULL, NULL) < 0){
            fprintf(stderr, "Failed to select fifos\n");
            clean_up(fifos, argv, argc);
            return EXIT_FAILURE;
        }

        for (int i = 0; i < clients; i++){

            if (FD_ISSET(fifos[i], &active_clients)){
                char buf[PIPE_BUF];
                read(fifos[i], buf, sizeof(buf));

                if (buf[0] == '#'){
                    strncpy(reg[i], buf + 1, PIPE_BUF);
                    printf("%s joined the server\n", reg[i]);
                    continue;
                }
                if (buf[0] == '\n'){
                    clients--;
                    FD_CLR(fifos[i], &tmpset);
                    FD_CLR(fifos[i], &active_clients);
                    printf("%s left the server\n", reg[i]);
                    continue;
                }
                printf("%s: %s", reg[i], buf);
            }
        }
        memcpy(&active_clients, &tmpset, sizeof(fd_set)); // <-- I added this line
    }
    clean_up(fifos, argv, argc);

    return EXIT_SUCCESS;
}

void clean_up(int* fifos, char** argv, int n){
    for (int i = 1; i < n; i++){
        close(fifos[i - 1]);
        close_fifo(argv[i]);
    }
}

And the client side is implemented like this:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include "fifo.h"

int main(int argc, char** argv){
    int fp = open_fifo(argv[argc - 1], O_WRONLY);

    char id[BUF_SIZE] = "";
    char* client_name = argv[argc - 1];

    strncat(id, "#", 1);
    strncat(id, client_name, strlen(client_name));

    write(fp, id, strlen(id) + 1);

    while (1){
        char buf[PIPE_BUF];

        printf("Message: ");
        fgets(buf, PIPE_BUF, stdin);
        write(fp, buf, strlen(buf) + 1);

        if (buf[0] == '\n'){
            break;
        }
    }
    close(fp);

    return EXIT_SUCCESS;
}

As far as I figured it out, it has something to do with the select command. It blocks until one of the fifos has some input ready, but only for the first client that sends a message. Unfortunately it does not unblock for the other clients. Additionally, I don't know whether my argument passed for nfds on the select command is the right one. I did also try to put in FD_SETSIZE but that didn't change much. Does anybody know what I am doing wrong here?