CODEWITHSUNDEEP
awk
Gustavo Alberto Gonzalez
Gustavo Alberto Gonzalez

Reputation: 3

Data processing AWK

I have created a shell script which filters certain data based on the input file that I write in the console. But now I have approximately like 30 files to be filtered and I was wondering if I could write all these files (PATH) that needed to be filtered in an .txt file and invoke it in the main awk file to filter each one of them without typing one by one in the console.

Regards.

awk -f "filter.awk" data.txt

data.txt should have file names: prueba1.tr prueba2.tr . . prueba30.tr

The filter in awk should read each one of the files inside data.txt. If there is a match then it proceeds to open the file a process the data inside. At the moment to do this I am just using if(ARGV[1]=="name of the file.tr") {proceed with the filtering process} But I have approximately 30 files and I would like to automatize this a bit.

Thanks for the suggestions

Upvotes: 0

Views: 263

Answers (3)

karakfa
karakfa

Reputation: 67467

awk can take multiple input files and does the processing one by one. If you want to have the output collected in the same file, it's a typical awk use case.

For example, if you ran your script as

awk -f script.awk inputfile > outputfile

You can extend to multiple files

awk -f script.awk inputfile1 inputfile2 ... > outputfile

or, if using standard naming

awk -f script.awk inputfile{1..30} > outputile

or, all your files are in a directory

awk -f script.awk inputfiles/* > outputile

etc,

If you want one output per input file, you need redirection in the script. Since we don't know what your script is I'm going to present a simple case.

For example, this will print the headers (first lines) of the input files indexed 1 to 30 to corresponding output files 

awk 'FNR==1{print > FILENAME".out"}' inputfiles{1..30}

Upvotes: 1

Ed Morton
Ed Morton

Reputation: 203169

$ cat file1
foo
$ cat file2
bar
$ cat data.txt
file1
file2

$ awk 'NR==FNR{ARGV[ARGC]=$0; ARGC++; next} {print FILENAME, $0}' data.txt
file1 foo
file2 bar

or if you prefer (a bit more efficient since NR==FNR isn't getting tested for every line of every file):

$ awk -v list="data.txt" 'BEGIN{ while ( (getline line < list) > 0 ) { ARGV[ARGC]=line; ARGC++ } close(list) } {print FILENAME, $0}'
file1 foo
file2 bar

Upvotes: 2

Carlos Bribiescas
Carlos Bribiescas

Reputation: 4397

Yeah, if files.txt has the list of your files, and yourCommand.shis your script that accepts a file as a parameter, you can do

cat files.txt | xargs -L 1 yourCommand.sh

That will feed in each line of files.txt to yourCommand.sh one by one. It doesn't use awk, but I think it accomplishes what you wanted.

Note: I don't care that its a 'useless use of cat'

Upvotes: 1

Related Questions