Output of command to array not working

Question

I'm attempting to store the output of a series of beeline HQL queries into an array, so that I can parse it to pull out the interesting bits. Here's the relevant code:

    #!/usr/bin/env ksh
    ext_output=()
    while IFS= read -r line; do
      ext_output+=( "$line" )
    done < <( bee  --hiveconf hive.auto.convert.join=false  -f temp.hql)

bee is just an alias to the full beeline command with the JDBC url, etc. Temp.hql is multiple hql queries.

And here's a snippet of what the output of each query looks like:

    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
    | tableName:myTable                                                                                                                                                                                                                                                                                                                                                                                                   |
    | owner:foo                                                                                                                                                                                                                                                                                                                                                                                                        |
    | location:hdfs://<server>/<path>...
<big snip>
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
    15 rows selected (0.187 seconds)

The problem is, my array is only getting the last line from each result (15 rows selected (0.187 seconds).

Am I doing something wrong here? The exact same approach is working in other instances, so I really don't understand.

Answer 1

Hmmmm, I'm not having any problems with the code you've posted.

I can reproduce what I think you may be seeing (ie, array contains a single value consisting of the last line of output) if I make the following change in your code:

# current/correct code - from your post
ext_output+=( "$line" )

# modified/wrong code
ext_output=+( "$line" )

Notice the placement of the plus sign (+):

• when on the left side of the equal sign (+=) each $line is appended to the end of the array (see sample run - below)
• when on the right side of the equal sign (=+) each $line is assigned to the first slot in the array (index=0); the plus sign (+) and parens (()) are treated as part of the data to be stored in the array (see sample run - at bottom of this post)

Could there be a typo between what you're running (with 'wrong' results) vs what you've posted here in this thread (and what you've mentioned generates the correct results in other instances)?

---

Here's what I get when I run your posted code (plus sign on the left of the equal sign : +=) ...

NOTE: I've replaced the bee/HCL call with an output file containing your sample lines plus a couple (bogus) data lines; also cut down the longer lines for readability:

$ cat temp.out
    -----------------------------------------+--+
    | tableName:myTable
    | owner:foo
    | location:hdfs://<server>/<path>...

abc def ghi
123 456 789

    -----------------------------------------+--+
    15 rows selected (0.187 seconds)

Then I ran your code against temp.out:

ext_output=()

while IFS= read -r line
do
    ext_output+=( "$line" )
done < temp.out

Some stats on the array:

$ echo "array size : ${#ext_output[*]}"
array size : 10

$ echo "array indx : ${!ext_output[*]}"
array indx : 0 1 2 3 4 5 6 7 8 9

$ echo "array vals : ${ext_output[*]}"
array vals :     -----------------------------------------+--+     | tableName:myTable                              | owner:foo                                      | location:hdfs://<server>/<path>...  abc def ghi 123 456 789      -----------------------------------------+--+     15 rows selected (0.187 seconds)

And a dump of the array's contents:

$ for i in ${!ext_output[*]}
> do
> echo "${i} : ${ext_output[$i]}"
> done
0 :     -----------------------------------------+--+
1 :     | tableName:myTable
2 :     | owner:foo
3 :     | location:hdfs://<server>/<path>...
4 :
5 : abc def ghi
6 : 123 456 789
7 :
8 :     -----------------------------------------+--+
9 :     15 rows selected (0.187 seconds)

---

If I modify your code to place the plus sign on the right side of the equal sign (=+) ...

ext_output=()

while IFS= read -r line
do
    ext_output=+( "$line" )
done < temp.out

... the array stats:

$ echo "array size : ${#ext_output[*]}"
array size : 1

$ echo "array indx : ${!ext_output[*]}"
array indx : 0

$ echo "array vals : ${ext_output[*]}"
array vals : +(     15 rows selected (0.187 seconds) )

... and the contents of the array:

$ for i in ${!ext_output[*]}
> do
> echo "${i} : ${ext_output[$i]}"
> done
0 : +(     15 rows selected (0.187 seconds) )

!! Notice that the plus sign and parens are part of the string stored in ext_output[0]