CODEWITHSUNDEEP
grep
Ben Dunkle
Ben Dunkle

Reputation: 320

Create a list of all matched strings without duplicates

I have a list of urls that look like:

http://example.com/page1
http://example.com/page1
http://example.com/page1
http://example.com/page2
http://example.com/page2
http://example.com/page3

From this, I want to create a list that's like:

http://example.com/page1
http://example.com/page2
http://example.com/page3

So if there are more than one match, I want to return only one of the matches. What would the grep pattern be for that? Thanks.

Upvotes: 0

Views: 34

Answers (1)

nu11p01n73R
nu11p01n73R

Reputation: 26667

You can very easily do it using awk

$ awk '!url[$0]++' input
http://example.com/page1
http://example.com/page2
http://example.com/page3

Upvotes: 1

Related Questions