CODEWITHSUNDEEP
linuxbashshellcommand-line
xdhmoore
xdhmoore

Reputation: 9925

How do you filter out all unique lines in a file?

Is there a way to filter out all unique lines in a file via commandline tools without sorting the lines? I'd like to essentially do this:

sort -u myFile

without the performance hit of sorting.

Upvotes: 5

Views: 2865

Answers (1)

Kent
Kent

Reputation: 195289

Remove duplicated lines:

awk '!a[$0]++' file

This is famous awk one-liner. there are many explanations on inet. Here is one explanation:

This one-liner is very idiomatic. It registers the lines seen in the associative-array "a" (arrays are always associative in Awk) and at the same time tests if it had seen the line before. If it had seen the line before, then a[line] > 0 and !a[line] == 0. Any expression that evaluates to false is a no-op, and any expression that evals to true is equal to "{ print }".

Upvotes: 20

Related Questions