CODEWITHSUNDEEP
bashshellcsvunixawk
blackbaddl
blackbaddl

Reputation: 62

Parsing multiple CSV files in bash by pattern with counter

Let's assume I have the files

I get the last value of second column with a little formatting

 awk -v max=0 '{if($1>max){want=$2; max=$1}}END{print "x:";print want} ' report-x-1.csv >> test.txt

How do I do this with any number of .csv files and have a counter on top?

awk -v max=0 '{if($1>max){want=$2; max=$1}}END{print "x:";print want} ' report-x-* >> test.txt

is only doing something for the first file.

What I need in the end is a .csv like this but with any number of inputfile:

x1 , 300
x2 , 250
x3 , 300
y1 , 270
y2 , 250
y3 , 280

EDIT:
I have not yet tried every idea in detail, to make it clearer here are two example files

https://pastebin.com/VMb32ULc

https://pastebin.com/qXQnagQ9

These files are could be named report-x-1.csv and report-y-1.csv and the output file should contain two lines x-1: 156. ... and y-1: 300. .... (which is the last line second column)
The solution should work with multiple input files like mentioned above.

Upvotes: 0

Views: 476

Answers (3)

RavinderSingh13
RavinderSingh13

Reputation: 133680

Could you please try following. Since no samples are given so couldn't test it. But this should be faster than a for loop which traverse through all csv files and calls awk in each iteration.

Following are the points taken care in this program:

  • NO need to use a for loop to traverse through .csv files, since awk is capable of it.
  • OP's code is NOT taking care of getting x, y values from file names I have added that logic too.
  • One could setup the output file name in BEGIN section of code as per need too.
awk -v max=0 '
BEGIN{
  OFS=" , "
  output_file="output.txt"
}
FNR==1{
  if(want){
    print output":"ORS want > (output_file)    
  }
  split(FILENAME,array,"[-.]")
  output=array[2] array[3]
  want=max=""
}
{
  if($1>max){
    want=$2
    max=$1
  }
}
END{
  print output":"ORS want > (output_file)
}
' *.csv

Typo fixed by OP

Upvotes: 2

KamilCuk
KamilCuk

Reputation: 141573

To run a script for each file in a directory you can use pipep the filenames and use xargs -n1 :

echo report-x-*.csv | xargs -n1 awk -v max=0 '{if($1>max){want=$2; max=$1}}END{print "x:";print want}' >> test.txt

Upvotes: 0

EdYuTo
EdYuTo

Reputation: 6874

Maybe trying something like:

for file in $(ls)
do
    echo 'Doing something to file:' $file
    awk -v max=0 '{if($1>max){want=$2; max=$1}}END{print "x:";print want} ' $file >> test.txt
done

Or maybe:

for file in $(ls)
do
    if [[ $file == *.csv ]]; then
        echo 'Doing something to file:' $file
        awk -v max=0 '{if($1>max){want=$2; max=$1}}END{print "x:";print want} ' $file >> test.txt
    fi
done

Upvotes: 2

Related Questions