How to concatenate and loop through two columns using just Bash variables, i.e. without temporary files

Question

I have two Bash variables that contain 2 columns of data. I'd like concatenate them to create two larger columns, and then use this outcome to loop in the resulting rows, having each column read in respective temporal variables.

I'll explain what I need with minimal working example. Let's think I have a tmp file with the following sample content:

for i in `seq 1 10`; do echo foo $i; done > tmp 
for i in `seq 1 10`; do echo bar $i; done >> tmp
for i in `seq 1 10`; do echo baz $i; done >> tmp

What I need is effectively equivalent to the following code that relies in external temporary files:

grep foo tmp > file1
grep bar tmp > file2

cat file1 file2 > file_tmp

while read word number
do
  if [ $word = "foo" ]
    then
    echo word $word number $number
  fi  
done < file_tmp


rm file1 file2 file_tmp

My question then is: how can I to achieve this result, i.e. concatenating the two columns and then looping across rows, without having to write out the temporary files file1, file2 and file_tmp?

Answer 1

Splitting then merging standard input in one operation

Of course, this could be used on standard input like output of any command, as well as on a file.

This demonstration use command output directly, without the requirement of temporary file.

First, the bunch of lines:

I've condensed your 1st tmp file into this one line command:

 . <(printf 'printf "%s %%d\n" {1..10};' foo bar baz)

For reducing output on SO, here is a sample of output for 3 lines by word (rest of this post will still use 10 values per word.):

. <(printf 'printf "%s %%d\n" {1..3};' foo bar baz)
foo 1
foo 2
foo 3
bar 1
bar 2
bar 3
baz 1
baz 2
baz 3

You will need a fifo for the split:

mkfifo $HOME/myfifo

Note: this could be done by using unnamed fifo (aka without temporary fifo), but you have to manage openning and closing file descriptor by your script.

tee for splitting, then paste for merging output:

Quick run:

. <(printf 'printf "%s %%d\n" {1..10};' foo bar baz) |
  tee >(grep foo  >$HOME/myfifo ) | grep ba  |
  paste -d $'\1' $HOME/myfifo - - | sed 's/\o1/ and /g'

(Last sed is just for cosmetic) This should produce:

foo 1 and bar 1 and bar 2
foo 2 and bar 3 and bar 4
foo 3 and bar 5 and bar 6
foo 4 and bar 7 and bar 8
foo 5 and bar 9 and bar 10
foo 6 and baz 1 and baz 2
foo 7 and baz 3 and baz 4
foo 8 and baz 5 and baz 6
foo 9 and baz 7 and baz 8
foo 10 and baz 9 and baz 10

With some bash script in between:

. <(printf 'printf "%s %%d\n" {1..10};' foo bar baz) | (
    tee >(
        while read -r word num;do
            case $word in
                foo ) echo Word: foo num: $num ;;
                * ) ;;
            esac
        done >$HOME/myfifo
      ) |
        while read -r word num;do
            case $word in
                ba* ) ((num%2))&& echo word: $word num: $num ;;
                * ) ;;
            esac
        done
  ) | paste $HOME/myfifo -

Should produce:

Word: foo num: 1        word: bar num: 1
Word: foo num: 2        word: bar num: 3
Word: foo num: 3        word: bar num: 5
Word: foo num: 4        word: bar num: 7
Word: foo num: 5        word: bar num: 9
Word: foo num: 6        word: baz num: 1
Word: foo num: 7        word: baz num: 3
Word: foo num: 8        word: baz num: 5
Word: foo num: 9        word: baz num: 7
Word: foo num: 10       word: baz num: 9

Other syntax, same job:

paste $HOME/myfifo <(
  . <(printf 'printf "%s %%d\n" {1..10};' foo bar baz) | 
  tee >(
    while read -r word num;do
        case $word in
            foo ) echo Word: foo num: $num ;;
            * ) ;;
        esac
    done >$HOME/myfifo
  ) |
    while read -r word num;do
        case $word in
            ba* ) ((num%2))&& echo word: $word num: $num ;;
            * ) ;;
        esac
    done
)

Removing fifo

rm $HOME/myfifo

Answer 2

you mean like this ??

paste <( jot - 1 9 2 ) <( jot - 2 10 2 )

1   2
3   4
5   6
7   8
9   10

Answer 3

I may have misunderstood, but if you want to do this without temp files, perhaps this would work for your use-case:

# Gather the output from the 3 'seq' commands and pipe into AWK
{ 
  for i in $(seq 1 10); do echo foo "$i"; done ;
  for i in $(seq 1 10); do echo bar "$i"; done ;
  for i in $(seq 1 10); do echo baz "$i"; done ; 
} |\
awk '{
  if ($1=="foo" || $1=="bar") {a[NR]=$1; b[NR]=$2}} 
  END{for (i in a) {print "word " a[i] " number " b[i]}
}'

# For the AWK command: if a line contains "foo" or "bar",
# create an array "a" for the word, indexed using the row number ("NR")
# and an array "b" for the number, indexed using the row number ("NR")
# Then print the arrays with the words "word" and "number" and the correct spacing

Result:

word foo number 1
word foo number 2
word foo number 3
word foo number 4
word foo number 5
word foo number 6
word foo number 7
word foo number 8
word foo number 9
word foo number 10
word bar number 1
word bar number 2
word bar number 3
word bar number 4
word bar number 5
word bar number 6
word bar number 7
word bar number 8
word bar number 9
word bar number 10

Answer 4

• Bash's read can take input from a file descriptor other than stdin.
• Bash has process substitution

while
    read -u3 foo1 foo2 &&
    read -u4 bar1 bar2
do
    echo "$foo1 $foo2 - $bar1 $bar2"
done 3< <(grep ^foo tmp) 4< <(grep ^bar tmp)

The code above is a kind of zip function. Note that it doesn't address ensuring that the ordering of the two sequences is correct.

---

It's not clear why your code in the question creates and then ignores bar lines. If you are doing that, the code is even simpler:

while read word number; do
    echo "word $word number $number"
done < <(grep ^foo tmp)

Answer 5

You use awk to achieve this.

awk '{if($1=="foo") {print "word "$1" number "$2}}' file_tmp