CODEWITHSUNDEEP
awkgrepduplicates
user1844845
user1844845

Reputation: 321

How can I delete duplicate words in text file

I have text file:

abandonovať
abandonovať, neabandonovať
abandonovať, abandonujem
abandonovať, abandonuješ
abandonovať, abandonuje
abandonovať, abandonuje
abandonovať, abandonuje
abandonovať, neabandonujem
abandonovať, neabandonuješ

And I want remove every duplicate word. It will look like this:

abandonovať
neabandonovať
abandonujem
abandonuješ
abandonuje
neabandonujem
neabandonuješ

Have somebody any idea how to make it in GREP, AWK... ?

Upvotes: 2

Views: 3993

Answers (3)

user1844845
user1844845

Reputation: 321

Thank you guys, In GREP it works but it sort words by alphabet. In AWK it also works, but I have one more little problem. My text file is:

abandonovať
abandonovať, neabandonovať
abandonovať, abandonujem
.
.
.

If I enter your code in awk my result is:

abandonovať
abandonovať,
neabandonovať
abandonujem
.
.
.

One word is there twice. Once with "," and once without ",".

Upvotes: 0

Steve
Steve

Reputation: 54402

Here's one way using GNU awk:

awk -F "[, ]" '{ for(i=1;i<=NF;i++) if (!a[$i]++ && $i != "") print $i }' file

Results:

abandonovať
neabandonovať
abandonujem
abandonuješ
abandonuje
neabandonujem
neabandonuješ

Upvotes: 5

lenik
lenik

Reputation: 23528

you may try to use:

grep -o '\w*' a.txt | sort | uniq

where a.txt is your file.

Upvotes: 5

Related Questions