CODEWITHSUNDEEP
linuxbashtextawksed
Oswald Roswell
Oswald Roswell

Reputation: 65

Use awk or sed to eliminate repeat entries in text file

I have a text file like this...

apples
berries
berries
cherries

and I want it to look like this...

apples
berries
cherries

That's it. I just want to eliminate doubled entries. I would prefer for this to be an awk or sed "one-liner" but if there's some other common bash tool that I have overlooked that would be fine.

Upvotes: 0

Views: 138

Answers (2)

user000001
user000001

Reputation: 33327

There is a special command for this task, called uniq:

$ uniq file
apples
berries
cherries

This requires that common lines are adjacent, not adjacent equal lines are not removed.

Upvotes: 2

Guru
Guru

Reputation: 16984

sort -u file

if in case you are not worried about the order of the output.

Remove duplicates by retaining the order:

awk '!a[$1]++' file

Upvotes: 5

Related Questions