CODEWITHSUNDEEP
jsonbashshelljq
Yassine Boumajane
Yassine Boumajane

Reputation: 21

Remove duplicate JSON blocks from file using JQ

I have a JSON file that contains thousands of entries, and i need to remove the duplicate blocks.

Here is an example of the file:

{ "signatures": [
   {
     "signatureId": 0050,
     "mode": 0
   },
   {
     "signatureId": 0012,
     "mode": 0
   },
   {
     "signatureId": 0012,
     "mode": 1
   }
]}

Here is the target result to achieve:

{ "signatures": [
   {
     "signatureId": 0050,
     "mode": 0
   },
   {
     "signatureId": 0012,
     "mode": 0
   }
]}

And as you see, the "mode" value doesnt matter, what really matters is that the "signatureId" must not be duplicate, so when we remove the whole block, which ever "mode" stays, its not a problem.

I can only use Shell and/or JQ.

Upvotes: 2

Views: 108

Answers (1)

pmf
pmf

Reputation: 36276

Use unique_by with the field to be checked for duplicates as its argument. It will always take the first of a kind (here, the one with "mode": 0)

jq '.signatures |= unique_by(.signatureId)'

{
  "signatures": [
    {
      "signatureId": 12,
      "mode": 0
    },
    {
      "signatureId": 50,
      "mode": 0
    }
  ]
}

Demo

Upvotes: 5

Related Questions