CODEWITHSUNDEEP
arraysjsonbashjq
Waffy
Waffy

Reputation: 477

Merge Arrays of JSON

So my objective is to merge json files obtain this format:

{
  "title": "NamesBook",
  "list": [
    {
      "name": "Ajay"
    },
    {
      "name": "Al"
    }
  ]
}

And I have files that look like this format:

blahblah.json

{
  "title": "NamesBook",
  "list": [
    {
      "name": "Ajay"
    }
  ]
}

blueblue.json

{
  "title": "NamesBook",
  "list": [
    {
      "name": "Al"
    }
  ]
}

I can store the list array of all my names in a variable with the following:

x = jq -s '.[].list' *.json

And then I was planning on appending the variable to an empty array in a file I created, out.json, which looks like this:

{
  "type": "NamesBook",
  "list": []
}

However, when my script runs over the line

jq '.list[] += "$x"' out.json'

It brings up a jq error:

Cannot iterate over null.

Even when I add a random element, the same error shows up. Tips on how I should proceed? Are there other tools in jq to help achieve merging arrays?

Upvotes: 44

Views: 61718

Answers (4)

Jeff Mercado
Jeff Mercado

Reputation: 134811

Assuming every file will have the same title and you're simply combining the list contents, you could do this:

$ jq 'reduce inputs as $i (.; .list += $i.list)' blahblah.json blueblue.json

This just takes the first item and adds to its list, the list of all the other inputs.

Since it appears this question got hijacked to answer a different problem... might as well offer answers to that too...

Given a stream of arrays, to combine the contents to a single array, I would write it:

$ jq -n '[inputs[]]' input.json
$ jq '[.[],inputs[]]' input.json

I wouldn't use slurp (-s) if I don't need to. Performance difference may be negligible, but is an unnecessary step. Though with larger inputs, you wouldn't want to use it anyway.

In such cases, you'll want to stream it in and you could filter out the ends of the arrays (the single item arrays).

$ jq --stream -n '[inputs|select(length==2)[1]]' input.json

Upvotes: 15

Jannis Ioannou
Jannis Ioannou

Reputation: 2191

Let me also provide just what the title asks for, because I'm sure a lot of people that stepped on this question look for something simpler.

Any of the following (added math2001 and pmf answers):

echo -e '["a","b"]\n["c","d"]' | jq -s 'add'

echo -e '["a","b"]\n["c","d"]' | jq -s 'flatten(1)'

echo -e '["a","b"]\n["c","d"]' | jq -s 'map(.[])'

echo -e '["a","b"]\n["c","d"]' | jq -s '[.[][]]'

echo -e '["a","b"]\n["c","d"]' | jq '.[]' | jq -s

results in:

[
"a",
"b",
"c",
"d"
]

Note: Also any of the above can apply to arrays of objects.

Upvotes: 56

peak
peak

Reputation: 116680

The OP did not specify what should happen if there are objects for which .title is not "NamesBook". If the intent is to select objects with .title equal to "NamesBook", one could write:

map(select(.title == "NamesBook"))
| {title: .[0].title, list: map( .list ) | add}

This assumes that jq is invoked with the -s option.

Incidentally, add is the way to go here: simple and fast.

Upvotes: 4

zeppelin
zeppelin

Reputation: 9355

You can merge your files with add (jq 1.3+):

jq -s '.[0].list=[.[].list|add]|.[0]' *.json

or flatten (jq 1.5+):

jq -s '.[0].list=([.[].list]|flatten)|.[0]' *.json

[.[].list] - creates an array of all "list" arrays

 [
  [
    {
      "name": "Ajay"
    }
  ],
  [
    {
      "name": "Al"
    }
  ]
]

[.[].list]|flatten - flatten it (or .[].list|add - add all the arrays together)

[
  {
    "name": "Ajay"
  },
  {
    "name": "Al"
  }
]

.[0].list=([.[].list]|flatten)|.[0] - replace the first "list" with the merged one, output it.

{
  "title": "NamesBook",
  "list": [
    {
      "name": "Ajay"
    },
    {
      "name": "Al"
    }
  ]
}

Upvotes: 24

Related Questions