CODEWITHSUNDEEP
bash
cianius
cianius

Reputation: 2412

if output file from loop isn't empty, copy last line bash

I have a loop, in a bash script. It runs a programme that by default outputs a text file when it works, and no file if it doesn't. I'm running it a large number of times (> 500K) so I want to merge the output files, row by row. If one iteration of the loop creates a file, I want to take the LAST line of that file, append it to a master output file, then delete the original so I don't end up with 1000s of files in one directory. The Loop I have so far is: 

oFile=/path/output/outputFile_
oFinal=/path/output.final
for counter in {101..200}
do
    $programme  $counter -out $oFile$counter
    if [ -s $oFile$counter ] ## This returns TRUE if file isn't empty, right?
    then
        out=$(tail -1  $oFile$counter)  
        final=$out$oFile$counter 
        $final  >> $oFinal 
    fi
done

However, it doesn't work properly, as it seems to not return all the files I want. So is the conditional wrong?

Upvotes: 1

Views: 75

Answers (1)

glenn jackman
glenn jackman

Reputation: 247002

You can be clever and pass the programme a process substitution instead of a "real" file:

oFinal=/path/output.final
for counter in {101..200}
do
    $programme  $counter -out >(tail -n 1)
done > $oFinal

$programme will treat the process substitution as a file, and all the lines written to it will be processed by tail

Testing: my "programme" outputs 2 lines if the given counter is even

$ cat programme
#!/bin/bash
if (( $1 % 2 == 0 )); then
    {  
        echo ignore this line
        echo $1
    } > $2
fi
$ ./programme 101 /dev/stdout
$ ./programme 102 /dev/stdout
ignore this line
102

So, this loop should output only the even numbers between 101 and 200

$ for counter in {101..200}; do ./programme $counter >(tail -1); done
102
104
[... snipped ...]
198
200

Success.

Upvotes: 1

Related Questions