bash script for formatting piped output

Question

Okay, you asked for it - here is my whole code that I am trying to work with.

The help I need with is this: postprocauto.sh: a script that presents the data in a nice readable format.

I am supposed to take a cat of the autocsv file (sample below - full file is 160k) and then pipe it through various other scripts so in the end, you will have this command line:

cat autocsv | ./ prepprocauto.sh | ./BMW.sh | ./6cyl.sh | ./hwyfe.sh | ./postprocauto.sh

My problem is that I can get everything to run except the postprocauto.ch. I need to be able to do this:

Requirements for this script:
It must print header information after 20 lines of data. 

The breaks must have a least one empty line between them 
before printing another header. 

The last line of the output should print out the number of 
records that were processed.

An example of the output:

Year Eng. Disp. Cyl. City FE Hwy FE Model
2013 1.5 4     39 38       ILX
2013 2    4    24 35      ILX
2013 2.4 4     22 31      ILX
2013 2.4 4      22 31      TSX
2013 2.4 4      21 29      TSX
2013 3.5 6      19 28      TSX
2013 1.3 4      41 44      INSIGHT
2013 1.3 4      41 44      INSIGHT
2013 3.5 6      20 29     TL 2WD
2013 3.7 6      18 26     TL 4WD
2013 3.7    6      17 25     TL 4WD
There were 11 records processed.

the different scripts are below - if you want the full size autocsv, tell me where to put it - 520k file size

prepprocauto.sh

#!/bin/bash

while read x
do
    echo $x | awk -F',' ' { print $1":"$2":"$4":"$7":"$8":"$10":"$11":"$12":"$22":"$24}'

done

BMW.sh

#!/bin/bash

selection='BMW'

if [ ! $# -lt 1 ]; then
    selection=$1
fi


while read y; do

    model=$(echo $y | awk -F':'  '{print $2 }')

    if [ "$model" == "$selection" ]; then
            echo $y
    fi
done

6cyl.sh

#!/bin/bash

selection='6'

if [ ! $# -lt 1 ]; then
    selection=$1
fi


while read y; do

    model=$(echo $y | awk -F':'  '{print $5 }')

    if [ "$model" == "$selection" ]; then
            echo $y
    fi
done

hwyfe.sh

#!/bin/bash

selection='31'

if [ ! $# -lt 1 ]; then
    selection=$1
fi


while read y; do

    hwy=$(echo $y | awk -F':'  '{print $7 }')

    if [ "$hwy" -gt "$selection" ]; then
            echo $y
    fi
done

postprocauto.sh

output=$(awk -F ':' '{print $1 "\t" $4 "\t"$5"\t" $6 "\t" $7 "\t" $3}')
echo "Year  Eng. Disp Cyl  City FE  Hwy FE   Model"
echo "$output"

Contents of the autocsv file (cut down a lot)

2013,Audi,Audi,TT Roadster quattro,ADX,67,2,4,Auto(AM-S6),22,31,26,28.4068,42.25
79,33.3217,22.407,31.1674,25.6515,,TC,Turbocharged,AMS,Automated Manual- Selecta
ble (e.g. Automated Manual with paddles),6,Y,N,A,10,GP,Gasoline (Premium Unleade
d Recommended),MPG,N,,,,,,,2200,2200,2,2,1,Two Seaters,car,Vehicle Specific 5-cy
cle label,6/18/12,12113,,N,N,,,N,N,ENGINE CODE CDMA ONLY.,N,,Y,CONTINUOUS VARIAB
LE VALVE TIMING,N,,,,,,,,,,,,,,,,,,,,,,,,,,,GDI,Spark Ignition Direct Injection,
N,N,5W40,33.3,7,7,,DADXV02.03UA,5,600,
2013,BMW,BMW,Z4 sDrive28i,BMX,428,2,4,Auto(A8),22,33,26,27.9499,46.8923,34.1594,
21.9803,33.2305,25.9308,,TC,Turbocharged,A,Automatic,8,Y,N,R,10,GP,Gasoline (Pre
mium Unleaded Recommended),MPG,N,,,,,,,2200,2200,2,2,1,Two Seaters,car,Derived 5
-cycle label,7/24/12,11033,,N,N,,,N,N,,N,,Y,variable valve timing at inlet and o
utlet valves,Y,variable valve lift at inlet valves,,,,,,,,,,,,,,,,,,,,,,,,,,GDI,
Spark Ignition Direct Injection,,Y,0W30,34.3,7,7,,DBMXJ02.0N20,5,600,
2013,BMW,BMW,Z4 sDrive28i,BMX,429,2,4,Manual(M6),22,34,26,28.3664,48.0364,34.774
1,22.2841,34.0033,26.3746,,TC,Turbocharged,M,Manual,6,N,N,R,10,GP,Gasoline (Prem
ium Unleaded Recommended),MPG,N,,,,,,,2200,2200,2,2,1,Two Seaters,car,Derived 5-
cycle label,7/25/12,11092,,N,N,,,N,N,,N,,Y,variable valve timing at inlet and ou
tlet valves,Y,variable valve lift at inlet valves,,,,,,,,,,,,,,,,,,,,,,,,,,GDI,S
park Ignition Direct Injection,,Y,0W30,34.9,7,7,,DBMXJ02.0N20,5,600,
2013,BMW,BMW,Z4 sDrive35i,BMX,436,3,6,Auto(AM-S7),17,24,19,21.1097,32.8224,25.14
8,16.8973,23.5833,19.3682,,TC,Turbocharged,AMS,Automated Manual- Selectable (e.g
. Automated Manual with paddles),7,N,N,R,10,GP,Gasoline (Premium Unleaded Recomm
ended),MPG,N,,,,,,,3000,3000,2,2,1,Two Seaters,car,Derived 5-cycle label,8/7/12,
12492,,N,N,,,N,N,,N,,Y,variable valve timing at inlet and outlet valves,N,,,,,,,
,,,,,,,,,,,,,,,,,,,,GDI,Spark Ignition Direct Injection,,Y,0W30,25.3,4,4,,DBMXV0
3.054R,5,,3400
2013,BMW,BMW,Z4 sDrive35i,BMX,435,3,6,Manual(M6),19,26,21,23.3,36.6,27.855,18.54
45,26.1997,21.352,,TC,Turbocharged,M,Manual,6,N,N,R,10,GP,Gasoline (Premium Unle
aded Recommended),MPG,N,,,,,,,2700,2700,2,2,1,Two Seaters,car,Derived 5-cycle la
bel,8/7/12,11743,,N,N,,,N,N,,N,,Y,variable valve timing at inlet and outlet valv
es,N,,,,,,,,,,,,,,,,,,,,,,,,,,,GDI,Spark Ignition Direct Injection,,Y,0W30,28.1,
5,5,,DBMXV03.054R,5,,1900

Answer 1

You can use the modulo function to display a header every nth line like so:

$ seq 1 12 > moo 
$ awk < moo 'NR%4==0 { print "\nheading\n" } {print} 
             END{ print "total lines: " NR}'
1
2
3

heading

4
5
6
7

heading

8
9
10
11

heading

12
total lines: 12

Answer 2

Try using column -t

echo $output | column -t

Answer 3

Just quote the $output

echo "$output"

Additionally, you may want to get rid of redundant use of cat and do

awk -F ':' '{print $1 "\t" $4 "\t"$5"\t" $6 "\t" $7 "\t" $3}' ./temp.log

instead