Aryan Kushwaha
Aryan Kushwaha

Reputation: 105

Extract each URL with a unique combination of parameters

I have a list of URLs:

https://example.com/invite?id=12
https://example.com/invite?id=12&per=5675
https://example.com/invite?id=156&per=2290
https://example.com/invite?id=12309&per=37245
https://example.com/sudo?root=false&admin=true
https://example.com/invite?id=453&admin=false
https://example.com/invite?id=278&admin=true
https://example.com/invite?id=90&admin=false

And I want to extract those with a unique combination of parameters. So the expected output is:

https://example.com/invite?id=12
https://example.com/invite?id=12&per=5675
https://example.com/invite?id=90&admin=false
https://example.com/sudo?root=false&admin=true

How can I do this?

Upvotes: 0

Views: 94

Answers (1)

oguz ismail
oguz ismail

Reputation: 50750

Using GNU awk:

awk '!seen[gensub(/=[^&]*/,"","g")]++' file

Upvotes: 1

Related Questions