CODEWITHSUNDEEP
unixawkterminaluniq
user3288154
user3288154

Reputation: 23

Remove all lines from file with duplicate value in field, including the first occurrence

I would like to remove all the lines in my data file that contain a value in column 2 that is repeated in column 2 in other lines.

I've sorted by the value in column 2, but can't figure out how to use uniq for just the values in one field as the values are not necessarily of the same length.

Alternately, I can remove lines with the duplicate using an awk one-liner like 

awk -F"[,]" '!_[$2]++'

but this retains the line with the first incidence of the repeated value in col 2.

As an example, if my data is

a,b,c
c,b,a
d,e,f
h,i,j
j,b,h

I would like to remove ALL lines (including the first) where b occurs in the second column. Like this:

d,e,f
h,i,j

Thanks for any advice!!

Upvotes: 2

Views: 1437

Answers (3)

Mark Setchell
Mark Setchell

Reputation: 208077

Some different awk:

awk -F, '
   BEGIN {f=0}
   FNR==NR {_[$2]++;next}
   f==0 {
      f=1
      for(j in _)if(_[j]>1)delete _[j]
   }
   $2 in _
' file file

Explanation

The awk passes through the file twice - that's why it appears twice at the end. On the first pass (when FNR==NR) I count the number of times each column 2 appears in array _[]. At the end of the first pass, I then delete all elements of _[] where that element has been seen more than once. Then, on the second pass, I print lines whose second field appears in _[].

Upvotes: 0

jaypal singh
jaypal singh

Reputation: 77185

If the order is not important then the following should work: 

awk -F, '
!seen[$2]++ {
    line[$2] = $0
}
END { 
    for(val in seen)
        if(seen[val]==1) 
          print line[val]
}' file

Output

h,i,j
d,e,f

Upvotes: 1

rpax
rpax

Reputation: 4496

Solution with grep:

grep -v -E '\b,b,\b' text.txt

Content of the file:

$ cat text.txt 
a,b,c
c,b,a
d,e,f
h,i,j
j,b,h
a,n,b
b,c,f

$ grep -v -E '\b,b,\b' text.txt 
d,e,f
h,i,j
a,n,b
b,c,f

Hope it helps

Upvotes: 0

Related Questions