CODEWITHSUNDEEP
duplicates
apasajja
apasajja

Reputation: 606

What is automated way to remove text line if duplicated

For example (below), every line is unique except green car

blue car
red car
green car
black car
white car
yellow car
green car
brown car

All I want is not remove one line of green car... but I want remove both of green car..
in other words, if any text line is duplicated 2 times or more, remove all of them

Upvotes: 1

Views: 286

Answers (3)

suman
suman

Reputation: 1

a small groovy script

def list =["blue car","red car","green car","green car","green car","brown car"]
def value =0
def finalList =[]
list.each{
    value = list.count(it)
    finalList.add(it)
    if(value >1){
        finalList.remove(it)
    }
}
def file = new File("dulicateRemoved.txt")
file<<finalList

Upvotes: 0

Fr&#233;d&#233;ric Hamidi
Fr&#233;d&#233;ric Hamidi

Reputation: 263107

If your strings are in a file named data in the current directory, the following command line does what you want on *NIX systems:

$ sort < data | uniq -c | awk '{ if ($1 == 1) print }' | cut -d' ' -f8-

EDIT: If you want to save the results back to the input file, you can do:

$ sort < data | uniq -c | awk '{ if ($1 == 1) print }' \
> | cut -d' ' -f8- > data.new && mv data.new data

To process all files in the current directory, you can do:

$ for f in *; do sort < $f | uniq -c | awk '{ if ($1 == 1) print }' \
> | cut -d' ' -f8- > $f.new && mv $f.new $f; done

Upvotes: 2

awm
awm

Reputation: 6570

blue car
red car
black car
white car
yellow car
brown car

Edit: You could do this in php:

$a = array_count_values($mydata);
foreach ($mydata as $key=>$datum) {
  if ($a[$datum] > 1) unset($mydata[$key]);
}

Upvotes: 1

Related Questions