CODEWITHSUNDEEP
jsonjqjsonlines
giacecco
giacecco

Reputation: 741

How to use jq to output JSONL (one independent JSON object per line)

My request sounds trivial but I could not find a way to do it. I have as input an array of JSON objects:

[
    {
        "foo": 1,
        "bar": 2
    },
    {
        "foo": 3,
        "bar": 4
    },
    (...)
]

and I want as output the JSONL version of the same, aka one object per line, not an array:

    { "foo": 1, "bar": 2 }
    { "foo": 3, "bar": 4 }
    (...)

This is not the same as using --compact-output, as that would preserve the array and give me:

    [ { "foo": 1, "bar": 2 }, { "foo": 3, "bar": 4 }, (...) ]

Thank you in advance.

Upvotes: 50

Views: 29479

Answers (4)

americanflavour
americanflavour

Reputation: 11

Following this question/answer, noticed that many tools generating JSON (in my case I'm exporting from a database) use ambiguous line endings -- jq doesn't play nicely with making JSONL if there's strange carriage returns.

something like:

cat db.output.json| tr -d '\r\n' | jq -c '.[]'>db.output.jsonl

fixed the behavior where

jq -c '.[]'

from the accepted answer didn't create JSONL.

I'd leave just as a comment but my poor lil reputation won't allow it :(

Upvotes: 0

peak
peak

Reputation: 116690

The answer to the original question is to use the filter .[] together with the -c command-line option:

$ jq -c '.[]'

If the input file is very large (notably, if it is too large to fit into memory), it may be better to use jq's --stream command-line option, or a companion tool. If the preservation of numerical precision is important, then you may wish to consider jm, which is particularly easy to use: invoking jm without any parameters suffices. See the jq Cookbook: https://github.com/stedolan/jq/wiki/Cookbook

Upvotes: 66

peak
peak

Reputation: 116690

If the input array is too large to fit into memory, you can use jq's so-called "streaming parser".

Here is an illustration using a generic approach, that is, it makes no assumptions about the items in the top-level array:

$ echo '[{"foo":"bar"},99,null,{"foo":"baz"}]' |
  jq -cn --stream 'fromstream( inputs|(.[0] |= .[1:]) | select(. != [[]]) )'
{"foo":"bar"}
99
null
{"foo":"baz"}
$

Upvotes: 2

giacecco
giacecco

Reputation: 741

Found: it's

map(tostring) | reduce .[] as $item (""; . + $item + "\n")

You also need to use --raw-output.

Upvotes: -2

Related Questions