CODEWITHSUNDEEP
unixawkcommandcompare
pnb1
pnb1

Reputation: 43

Find duplicate words in two text files using command line

I have two text files:

f1.txt

boom Boom pow
Lazy dog runs.
The Grass is Green
This is TEST
Welcome

and

f2.txt

Welcome
I am lazy
Welcome, Green
This is my room
Welcome
bye

In Ubuntu Command Line I am trying:

awk 'BEGIN {RS=" "}FNR==NR {a[$1]=NR; next} $1 in a' f1.txt f2.txt

and getting output:

Green
This
is

My desired output is: 

lazy
Green
This is
Welcome

Description: I want to compare two txt files, line by line. Then I want to output all duplicate words. The matches should be not case sensitive. Also, comparing line by line would be better instead of looking for a match from f1.txt in a whole f2.txt file. In example, the word "Welcome" should not be in desired output if it was on line 6 instead of line 5 in f2.txt

Upvotes: 0

Views: 1519

Answers (1)

Wintermute
Wintermute

Reputation: 44023

Well, then. With awk:

awk 'NR == FNR { for(i = 1; i <= NF; ++i) { a[NR,tolower($i)] = 1 }; next } { flag = 0; for(i = 1; i <= NF; ++i) { if(a[FNR,tolower($i)]) { printf("%s%s", flag ? OFS : "", $i); flag = 1 } } if(flag) print "" }' f1.txt f2.txt

This works as follows:

NR == FNR {                                 # While processing the first file:
  for(i = 1; i <= NF; ++i) {                # Remember which fields were in
    a[NR,tolower($i)] = 1                   # each line (lower-cased)
  }
  next                                      # Do nothing else.
}
{                                           # After that (when processing the
                                            # second file)
  flag = 0                                  # reset flag so we know we haven't
                                            # printed anything yet
  for(i = 1; i <= NF; ++i) {                # wade through fields (words)
    if(a[FNR,tolower($i)]) {                # if this field was in the
                                            # corresponding line in the first
                                            # file, then
      printf("%s%s", flag ? OFS : "", $i)   # print it (with a separator if it
                                            # isn't the first)
      flag = 1                              # raise flag
    }
  }
  if(flag) {                                # and if we printed anything
    print ""                                # add a newline at the end.
  }
}

Upvotes: 2

Related Questions