CODEWITHSUNDEEP
jsonstreamjq
Josh Friedlander
Josh Friedlander

Reputation: 11657

Retrieving the first entity out of several ones

I am a rank beginner with jq, and I've been going through the tutorial, but I think there is a conceptual difference I don't understand. A common problem I encounter is that a large JSON file will contain many objects, each of which is quite big, and I'd like to view the first complete object, to see which fields exist, what types, how much nesting, etc.

In the tutorial, they do this:

# We can use jq to extract just the first commit.
$ curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[0]'

Here is an example with one object - here, I'd like to return the whole array (just like my_array=['foo']; my_array[0] would return foo in Python). 

wget https://hacker-news.firebaseio.com/v0/item/8863.json

I can access and pretty-print the whole thing with .

$ cat 8863.json | jq '.'
$
{
  "by": "dhouston",
  "descendants": 71,
  "id": 8863,
  "kids": [
    9224,
     ...
    8876
  ],
  "score": 104,
  "time": 1175714200,
  "title": "My YC app: Dropbox - Throw away your USB drive",
  "type": "story",
  "url": "http://www.getdropbox.com/u/2/screencast.html"
}

But trying to get the first element fails:

$ cat 8863.json| jq '.[0]' 
$ jq: error (at <stdin>:0): Cannot index object with number

I get the same error jq '.[0]' 8863.json, but strangely echo 8863.json | jq '.[0]' gives me parse error: Invalid numeric literal at line 2, column 0. What is the difference? Also, is this not the correct way to get the zeroth member of the JSON?

I've looked at other SO posts with this error message and at the manual, but I'm still confused. I think of the file as an array of JSON objects, and I'd like to get the first. But it looks like jq works with something called a "stream", and does operations on all of it (say, return one given field from every object).

Clarification:

Let's say I have 2 objects in my JSON:

{
  "by": "pg",
  "id": 160705,
  "poll": 160704,
  "score": 335,
  "text": "Yes, ban them; I'm tired of seeing Valleywag stories on News.YC.",
  "time": 1207886576,
  "type": "pollopt"
}
{
  "by": "dpapathanasiou",
  "id": 16070,
  "kids": [
    16078
  ],
  "parent": 16069,
  "text": "Dividends don't mean that much: Microsoft in its dominant years (when they had 40%-plus margins and were raking in the cash) never paid a dividend (they did so only recently).",
  "time": 1177355133,
  "type": "comment"
}

How would I get the entire first object (lines 1-9) with jq?

Upvotes: 1

Views: 2097

Answers (2)

peak
peak

Reputation: 116880

a large JSON file will contain many objects, each of which is quite big, and I'd like to view the first complete object, to see which fields exist, what types, how much nesting, etc.

As implied in @OguzIsmail's response, there are important differences between:

- a JSON file (i.e, a file containing exactly one JSON entity);
- a file containing a sequence (i.e., stream) of JSON entities; 
- a file containing an array of JSON entities.

In the first two cases, you can write jq -n input to select the first entity, and in the case of an array of entities, jq .[0] will suffice.

(In JSON-speak, a "JSON object" is a kind of dictionary, and is not to be confused with JSON entities in general.)

If you have a bunch of JSON objects (whether as a stream or array or whatever), just looking at the first often doesn't really give an accurate picture of all them. For getting a bird's eye view of a bunch of objects, using a "schema inference engine" is often the way to go. For this purpose, you might like to consider my schema.jq schema inference engine. It's usually very simple to use but of course how you use it will depend on whether you have a stream or array of JSON entities. For basic details, see https://gist.github.com/pkoppstein/a5abb4ebef3b0f72a6ed; for related topics (e.g. verification), see the entry for JESS at https://github.com/stedolan/jq/wiki/Modules

Please note that schema.jq infers a structural schema that mirrors the entities under consideration. Such structural schemas have little in common with JSON Schema schemas, which you might also like to consider.

Upvotes: 3

oguz ismail
oguz ismail

Reputation: 50795

Cannot index object with number

This error message says it all, you can't index objects with numbers. If you want to get the value of by field, you need to do

jq '.by' file

Wrt

echo 8863.json | jq '.[0]' gives me parse error: Invalid numeric literal at line 2, column 0.

It's normal since you didn't specify -R/--raw-input flag, and so jq sees the shell string 8863.json as a JSON string, and one cannot apply array indexing to JSON strings. (To get the first character as a string, you'd write .[0:1].)

If your input file consists of several separate entities, to get the first one:

jq -n 'input' file

or,

jq -n 'first(inputs)' file

To get nth (let's say 5th for example):

jq -n 'nth(5; inputs)' file

Upvotes: 3

Related Questions