CODEWITHSUNDEEP
linuxbashshellubuntu
iwahdan
iwahdan

Reputation: 449

I want to print 1 version of Duplicates and remove unique lines in bash linux

The Input is as Follows:

Mark
John
Marry
Mark
John

And output should be:

Mark
John

How could i do this in bash file linux.

Upvotes: 0

Views: 121

Answers (4)

Baba
Baba

Reputation: 850

perl onliner:

perl -ne '{$o{$_}++}END{for $i (keys %o){print $i if $o{$i}>1;}}'

Upvotes: 0

David C. Rankin
David C. Rankin

Reputation: 84559

Don't forget, you can do it in bash without any external tools as well:

#!/bin/bash

names=(`<dat/names.dat`)

sz=${#names[@]}

for ((i=0; i<$((sz-1)); i++)); do 

    for ((j=$((i+1)); j<$sz; j++)); do

        test "$i" -eq "$j" && continue
        test "${names[$i]}" == "${names[$j]}" && echo "${names[$i]}"

    done

done

output:

Mark
John

Upvotes: 0

that other guy
that other guy

Reputation: 123490

sort | uniq -d

For example:

$ cat file
Mark
John
Marry
Mark
John 

$ cat file | sort | uniq -d
John
Mark

To preserve order, you can use a more obscure awk command:

$ awk 'a[$0]++ == 1' file
Mark
John

Upvotes: 4

buff
buff

Reputation: 2053

To preserve order and still remove unique lines, you can use perl:

#!/usr/bin/perl

use strict;
use warnings;

my (%count, @lines);

$count{$_}++   || push @lines, $_ while <>;
$count{$_} > 1 && print         for @lines;

awk 'a[$0]++' file only preserves order of second occurences.

Upvotes: 0

Related Questions