How to create an anonymous pipe between 2 child processes and know their pids (while not using files/named pipes)?

Question

Please note that this questions was edited after a couple of comments I received. Initially I wanted to split my goal into smaller pieces to make it simpler (and perhaps expand my knowledge on various fronts), but it seems I went too far with the simplicity :). So, here I am asking the big question.

Using bash, is there a way one can actually create an anonymous pipe between two child processes and know their pids?

The reason I'm asking is when you use the classic pipeline, e.g.

cmd1 | cmd2 &

you lose the ability to send signals to cmd1. In my case the actual commands I am running are these

./my_web_server | ./my_log_parser &

my_web_server is a basic web server that dump a lot of logging information to it's stdout my_log_parser is a log parser that I wrote that reads through all the logging information it receives from my_web_server and it basically selects only certain values from the log (in reality it actually stores the whole log as it received it, but additionally it creates an extra csv file with the values it finds).

The issue I am having is that my_web_server actually never stops by itself (it is a web server, you don't want that from a web server :)). So after I am done, I need to stop it myself. I would like for the bash script to do this when I stop it (the bash script), either via SIGINT or SIGTERM.

For something like this, traps are the way to go. In essence I would create a trap for INT and TERM and the function it would call would kill my_web_server, but... I don't have the pid and even though I know I could look for it via ps, I am looking for a pretty solution :).

Some of you might say: "Well, why don't you just kill my_log_parser and let my_web_server die on its own with SIGPIPE?". The reason why I don't want to kill it is when you kill a process that's at the end of the pipeline, the output buffer of the process before it, is not flushed. Ergo, you lose stuff.

I've seen several solutions here and in other places that suggested to store the pid of my_web_server in a file. This is a solution that works. It is possible to write the pipeline by fiddling with the filedescriptors a bit. I, however don't like this solution, because I have to generate files. I don't like the idea of creating arbitrary files just to store a 5-character PID :).

What I ended up doing for now is this:

#!/bin/bash

trap " " HUP

fifo="$( mktemp -u "$( basename "${0}" ).XXXXXX" )"
mkfifo "${fifo}"

<"${fifo}" ./my_log_parser &
parser_pid="$!"

>"${fifo}" ./my_web_server &
server_pid="$!"

rm "${fifo}"

trap '2>/dev/null kill -TERM '"${server_pid}"'' INT TERM

while true; do
  wait "${parser_pid}" && break
done

This solves the issue with me not being able to terminate my_web_server when the script receives SIGINT or SIGTERM. It seems more readable than any hackery fiddling with file descriptors in order to eventually use a file to store my_web_server's pid, which I think is good, because it improves the readability.

But it still uses a file (named pipe). Even though I know it uses the file (named pipe) for my_web_server and my_log_parser to talk (which is a pretty good reason) and the file gets wiped from the disk very shortly after it's created, it's still a file :).

Would any of you guys know of a way to do this task without using any files (named pipes)?

Answer 1

From the Bash man pages:

!      Expands  to  the  process ID of the most recently executed back-
       ground (asynchronous) command.

You are not running a background command, you are running process substitution to read to file descriptor 3.

The following works, but I'm not sure if it is what you are trying to achieve:

sleep 120 &
child_pid="$!"

wait "${child_pid}"
sleep 120

Edit: Comment was: I know I can pretty much do this the silly 'while read i; do blah blah; done < <( ./my_proxy_server )'-way, but I don't particularly like the fact that when a script using this approach receives INT or TERM, it simply dies without telling ./my_proxy_server to bugger off too :)

So, it seems like your problem stems from the fact that it is not so easy to get the PID of the proxy server. So, how about using your own named pipe, with the trap command:

pipe='/tmp/mypipe'
mkfifo "$pipe"
./my_proxy_server > "$pipe" &

child_pid="$!"
echo "child pid is $child_pid"

# Tell the proxy server to bugger-off
trap 'kill $child_pid' INT TERM

while read
do
    echo $REPLY
    # blah blah blah
done < "$pipe"

rm "$pipe"

You could probably also use kill %1 instead of using $child_pid.

YAE (Yet Another Edit):
You ask how to get the PIDS from:

./my_web_server | ./my_log_parser &

Simples, sort of. To test I used sleep, just like your original.

sleep 400 | sleep 500 &
jobs -l

Gives:

[1]+  8419 Running                 sleep 400
      8420 Running                 | sleep 500 &

So its just a question of extracting those PIDS:

pid1=$(jobs -l|awk 'NR==1{print $2}')
pid2=$(jobs -l|awk 'NR==2{print $1}')

I hate calling awk twice here, but anything else is just jumping through hoops.