CODEWITHSUNDEEP
bashawk
Omri
Omri

Reputation: 1656

Improving a bash script for csv files

I have a bunch of CSV files in a folder. All of them on the same structure. more than 2k columns. The first column is ID.

I need to do the following for each file: For each n odd column (except the first column), do the following:

I have the following code:

for f in *.csv; do
        awk 'BEGIN { FS=OFS="," }
        NR==1 {
      for (i=3; i<=NF; i+=2)
         a[i]
     }FNR==NR {
           for (i=1; i<=NF; i++)
              sums[i] += $i;
           ++r;
           next
        } {
           for (i=1; i<=NF; i++)
              if (sums[i] > 0 && sums[i+1]>0 && sums[i] != 100*r)
                 printf "%s%s", (i>1)?OFS:"", $i;
              else print "removed index: " i > "removed.index"
              print ""
   }' "$f"  "$f" > "new_$f"
done

For some reason the ID column (first column) is been removed.

Input:

23232,0,0,5,0,1,100,3,0,33,100
21232,0,0,5,0,1,100,3,0,33,100
23132,0,0,5,0,1,100,3,0,33,100
23212,0,0,5,0,1,100,3,0,33,100
24232,0,0,5,0,1,100,3,0,33,100
27232,0,0,5,0,1,100,3,0,33,100

Current output (bad):

,1,33
,1,33
,1,33
,1,33
,1,33
,1,33

Expected output:

23232,1,33
21232,1,33
23132,1,33
23212,1,33
24232,1,33
27232,1,33

Can anyone check what is the issue?

Upvotes: 1

Views: 71

Answers (1)

anubhava
anubhava

Reputation: 785856

You need to skip first column from the logic to check for 0 in previous column:

awk 'BEGIN{FS=OFS=","; out=ARGV[1] ".removed.index"}
FNR==NR {
   for (i=1; i<=NF; i++)
      sums[i] += $i;
   ++r;
   next
} FNR==1 {
   for (i=3; i<=NF; i++) {
      if (sums[i] == 0) {
         if (i-1 in sums) {
            delete sums[i-1];
            print "removed index: " (i-1) > out
         }
         delete sums[i];
         print "removed index: " i > out
      } else if (sums[i] == 100*r) {
         delete sums[i];
         print "removed index: " i > out
      }
   }
} {
   printf "%s", $1
   for (i=2; i<=NF; i++)
      if (i in sums)
         printf "%s%s", OFS, $i;
   printf "%s", ORS
} END{close(out)}' file file

Output:

23232,1,33
21232,1,33
23132,1,33
23212,1,33
24232,1,33
27232,1,33

Also removed indices is:

cat file.removed.index

cat removed.index
removed index: 2
removed index: 3
removed index: 4
removed index: 5
removed index: 7
removed index: 8
removed index: 9
removed index: 11

Upvotes: 3

Related Questions