Stream specific numbered Bash file descriptor into variable

Question

I am trying to stream a specific numbered file descriptor into a variable in Bash. I can do this from normal standard in using the following function, but, how do it do it from a specific file descriptor. I need to direct the FD into the sub-shell if I use the same approach. I could always do it reading line by line, but, if I can do it in a continuous stream then that would be massively preferable.

The function I have is:

streamStdInTo ()
{
    local STORE_INvar="${1}" ; shift
    printf -v "${STORE_INvar}" '%s' "$( cat - )"
}

Yes, I know that this wouldn't work normally as the end of a pipeline would be lost (due to its execution in a sub-shell), however, either in the context of the Bash 4 set +m ; shopt -s lastpipe method of executing the end of a pipeline in the same shell as the start, or, by directing into this via a different file descriptor I am hoping to be able to use it.

So, my question is, How do I use the above but with different file descriptors than the normal?

Answer 1

I think there's a lot of confusion about what exactly you're trying to do. If I understand correctly the end goal here is to run a pipeline and capture the output in a variable, right? Kind of like this:

var=$(cmd1 | cmd2)

Except I guess the idea here is that the name of "$var" is stored in another variable:

varname=var

You can do an end-run around Bash's usual job control situation by using process substitution. So instead of this normal pipeline (which would work in ksh or zsh, but not in bash unless you set lastpipe):

cmd1 | cmd2 | read "$varname"

You would use this command, which is equivalent apart from how the shell handles the job:

read "$varname" < <(cmd1 | cmd2)

With process substitution, "read $varname" isn't run in a pipeline, so Bash doesn't fork to run it. (You could use your streamStdInTo() function there as well, of course)

As I understand it, you wanted to solve this problem by using numeric file descriptors:

cmd1 | cmd2 >&$fd1 &
read "$varname" <&$fd2

To create those file descriptors that connect the pipeline background job to the "read" command, what you need is called a pipe, or a fifo. These can be created without touching the file system (the shell does it all the time!) but the shell doesn't directly expose this functionality, which is why we need to resort to mkfifo to create a named pipe. A named pipe is a special file that exists on the filesystem, but the data you write to it doesn't go to the disk. It's a data queue stored in memory (a pipe). It doesn't need to stay on the filesystem after you've opened it, either, it can be deleted almost immediately:

pipedir=$(mktemp -d /tmp/pipe_maker_XXXX)
mkfifo ${pipedir}/pipe
exec {temp_fd}<>${pipedir}/pipe         # Open both ends of the pipe
exec {fd1}>${pipedir}/pipe
exec {fd2}<${pipedir}/pipe
exec {temp_fd}<&-                       # Close the read/write FD
rm -rf ${pipedir}                       # Don't need the named FIFO any more

One of the difficulties in working with named pipes in the shell is that attempting to open them just for reading, or just for writing causes the call to block until something opens the other end of the pipe. You can get around that by opening one end in a background job before trying to open the other end, or by opening both ends at once as I did above.

The "{fd}<..." syntax dynamically assigns an unused file descriptor number to the variable $fd and opens the file on that file descriptor. It's been around in ksh for ages (since 1993?), but in Bash I think it only goes back to 4.1 (from 2010).

Answer 2

It's not entirely clear what you mean, but perhaps you are looking for something like:

cat - <&4  # read from fd 4

Or, just call your current function with the redirect:

streamStdInTo foo <&4

edit: Addressing some questions from the comment, you can use a fifo:

#!/bin/bash

trap 'rm -f $f' 0
f=$(mktemp xxx)
rm $f
mkfifo $f
echo foo > $f &
exec 4< $f
cat - <&4
wait