CODEWITHSUNDEEP
perlsedawkgrep
coconut
coconut

Reputation: 1101

Counting lines ignored by grep

Let me try to explain this as clearly as I can...

I have a script that at some point does this: 

grep -vf ignore.txt input.txt

This ignore.txt has a bunch of lines with things I want my grep to ignore, hence the -v (meaning I don't want to see them in the output of grep).

Now, what I want to do is I want to be able to know how many lines of input.txt have been ignored by each line of ignore.txt.

For example, if ignore.txt had these lines: 

line1
line2
line3

I would like to know how many lines of input.txt were ignored by ignoring line1, how many by ignoring line2, and so on.

Any ideas on how can I do this?

I hope that made sense... Thanks!

Upvotes: 3

Views: 554

Answers (7)

David W.
David W.

Reputation: 107040

Are both ignore.txt and input.txt sorted?

If so, you can use the comm command!

$ comm -12 ignore.txt input.txt

How many lines are ignored?

$ comm -12 ignore.txt input.txt | wc -l

Or, if you want to do more processing, combine comm with awk.:

$ comm ignore.txt input.txt | awk '
    END {print "Ignored lines = " igtotal " Lines not ignored = "commtotal " Lines unique to Ignore file = " uniqtotal}
    {
       if ($0 !~ /^\t/) {uniqtotal+=1}
       if ($0 ~ /^\t[^\t]/) {commtotal+=1}
       if ($0 ~ /^\t\t/) {igtotal+=1}
    }'

Here I'm taking advantage with the tabs that are placed in the output by the comm command: * If there are no tabs, the line is in ignore.txt only. * If there is a single tab, it is in input.txt only * If there are two tabs, the line is in both files.

By the way, not all the lines in ignore.txt are ignored. If the line isn't also in input.txt, the line can't really be said to be ignored.

With Dennis Williamson's Suggestion

comm ignore.txt input.txt | awk '
   !/^\t/ {uniqtotal++}
   /^\t[^\t]/ {commtotal++}
   /^\t\t/ {igtotal++}
     END {print "Ignored lines = " igtotal " Lines not ignored = "commtotal " Lines unique to Ignore file = " uniqtotal}'

Upvotes: 0

potong
potong

Reputation: 58430

This might work for you:

# seq 1 15 | sed '/^1/!d' | sed -n '$='
7

Explanation:

Delete all lines except those that match. Pipe these matching (ignored) lines to another sed command. Delete all these lines but show the line number only of the last line. So in this example 1 thru 15, lines 1,10 thru 15 are ignored - a total of 7 lines.

EDIT:

Sorry misread the question (still a little confused!):

 sed 's,.*,sed "/&/!d;s/.*/matched &/" input.txt| uniq -c,' ignore.txt | sh

This shows the number of matches for each pattern in the the ignore.txt

 sed 's,.*,sed "/&/d;s/.*/non-matched &/" input.txt | uniq -c,' ignore.txt | sh

This shows the number of non-matches for each pattern in the the ignore.txt

If using GNU sed, these should work too:

sed 's,.*,sed "/&/!d;s/.*/matched &/" input.txt | uniq -c,;e' ignore.txt

or

sed 's,.*,sed "/&/d;s/.*/non-matched &/" input.txt | uniq -c,;e' ignore.txt

N.B. Your success with patterns may vary i.e. check for meta characters beforehand.

On reflection I thought this can be improved to:

sed 's,.*,/&/i\\matched &,;$a\\d' ignore.txt | sed -f - input.txt | sort -k2n | uniq -c

or

sed 's,.*,/&/!i\\non-matched &,;$a\\d' ignore.txt | sed -f - input.txt | sort -k2n | uniq -c

But NO, on large files this is actually slower.

Upvotes: 0

jmcnamara
jmcnamara

Reputation: 41584

This will print the number of ignored matches along with the matching pattern:

grep -of ignore.txt input.txt | sort | uniq -c

For example:

$ perl -le 'print "Coroline" . ++$s for 1 .. 21' > input.txt
$ perl -le 'print "line2\nline14"'               > ignore.txt

$ grep -of ignore.txt input.txt | sort | uniq -c
      1 line14
      3 line2

I.e., A line matching "line14" was ignored once. A line matching "line2" was ignored 3 times.

If you just wanted to count the total ignored lines this would work:

grep -cof ignore.txt input.txt

Update: modified the example above to use strings so that the output is a little clearer.

Upvotes: 0

sorpigal
sorpigal

Reputation: 26086

while IFS= read -r pattern ; do
        printf '%s:' "$pattern"
        grep -c -v "$pattern" input.txt
done < ignore.txt

grep with -c counts matching lines, but with -v added it counts non-matching lines. So, simply loop over the patterns and count once for each pattern.

Upvotes: 0

TLP
TLP

Reputation: 67908

This script will count the matched lines by hash lookup and save the lines to be printed in @result, where you may process them as you will. To emulate grep, just print them.

I made the script so it can print out an example. To use with the files, uncomment the code in the script, and comment the ones marked # example line.

Code:

use strict;
use warnings;
use v5.10;
use Data::Dumper;  # example line

# Example data. 
my @ignore = ('line1' .. 'line9'); # example line
my @input  = ('line2' .. 'line9', 'fo' .. 'fx', 'line2', 'line3'); # example line

#my $ignore = shift;  # first argument is ignore.txt
#open my $fh, '<', $ignore or die $!; 
#chomp(my @ignore = <$fh>);
#close $fh;

my @result;

my %lookup = map { $_ => 0 } @ignore;
my $rx = join '|', map quotemeta, @ignore;

#while (<>) {  # This processes the remaining arguments, input.txt etc
for (@input) { # example line
    chomp;     # Required to avoid bugs due to missing newline at eof
    if (/($rx)/) {
        $lookup{$1}++;
    } else {
        push @result, $_;
    }
}

#say for @result;       # This will emulate grep
print Dumper \%lookup;  # example line

Output:

$VAR1 = {
          'line6' => 1,
          'line1' => 0,
          'line5' => 1,
          'line2' => 2,
          'line9' => 1,
          'line3' => 2,
          'line8' => 1,
          'line4' => 1,
          'line7' => 1
        };

Upvotes: 0

tadmc
tadmc

Reputation: 3744

Note that the sum of the ignored lines plus the shown lines may NOT add up to the total number of lines... "line1 and line2 are here" will be counted twice.

#!/usr/bin/perl
use warnings;
use strict;

local @ARGV = 'ignore.txt';
chomp(my @pats = <>);

foreach my $pat (@pats) {
    print "$pat: ", qx/grep -c $pat input.txt/;
}

Upvotes: 1

Bubu
Bubu

Reputation: 173

According to unix.stackexchange 

grep -o pattern file | wc -l

counts the total number of a given pattern in the file. A solution, given this and the information, that you already use a script, is to use several grep instances to filter and count the patterns, which you want to ignore.

However, I'd try to build a more comfortable solution involving a scripting language like e.g. python.

Upvotes: 0

Related Questions