CODEWITHSUNDEEP
jsonshelljqfifo
kyb
kyb

Reputation: 8201

Take only first json object from stream with jq, do not touch rest

This thread Split multiple input JSONs with jq helped me to solve one problem. But not other.

mkfifo xxs
exec 3<>xxs  ## keep open file descriptor
echo '{"a":0,"b":{"c":"C"}}{"x":33}{"asd":889}' >&3
jq -nc input <&3  ## prints 1st object '{"a":0,"b":{"c":"C"}}' and reads out the rest
cat <&3 ## prints nothing

My problem is to make jq stop reading after first object is read, and do not touch other data in stream (fifo). So cat should show the rest of data: '{"x":33}{"asd":889}'.

How to achive that with jq?

Upvotes: 4

Views: 1353

Answers (2)

bigearsbilly
bigearsbilly

Reputation: 24

If you want to partially read a stream you will probably need to do it yourself. You could write a trivial C program to do this. I doubt there are any off-the-shelf parsers you can find to specify stopping the read of a stream after n objects. As mentioned before, most stream readers will use stdio and read all they can into a buffer.

Upvotes: 0

Alex Jasmin
Alex Jasmin

Reputation: 39506

jq doesn't have to read to the whole input to get the first value. This can be verified by feeding an infinite sequence of values to jq which takes the first value and exit:

yes '{}' | jq -n input

Though, the question assumes a bit more. Namely that jq can read a single JSON value from a named pipe and stop reading "right at that point" so the rest can be then read by cat.

mkfifo xxs
exec 3<>xxs              ## keep open file descriptor
echo '1 2 3' >&3
jq -nc input <&3 >first  ## Get first value
cat <&3 >rest            ## Nothing to show jq read all data

This gets more complicated as we don't know where that first value ends and most Unix programs (jq included) read input in larger chunks to limit the number of read syscalls.

jq would need an option to read its input one byte at a time. And, while this could be implemented, it may be of limited utility.

The closest thing I can think of is to output the first value to stderr and the rest to stdout.

jq -n 'input | stderr | inputs' <&3 2>first 1>rest

Input is processed in a streaming fashion (one input value at a time) and you can pipe stdout and/or stderr to something else. Though the whole input has to be valid JSON and it will be prettified while passing through jq (unlike with cat above).

If reading from a named pipe is not a requirement and you can afford to read the input from a file. Then, you can access the first value and the rest in two separate invocations.

echo '1 2 3' > in
jq -n 'input' in >first
jq -n 'input | inputs' in >rest

If stream processing is the goal, it may also be possible to do everything in a single jq script that processes its input incrementally.

This all assumes top-level values. Though, jq can also process nested structures incrementally using the --stream option.

Upvotes: 2

Related Questions