CODEWITHSUNDEEP
pythonjson
lordy
lordy

Reputation: 630

Large ndjson file can not be loaded correctly in Python

I have a json file with a size of 5 GB. I would like to load it and do some EDA on it in order to figure out where the relevant information is.

I tried:

import json
import pprint

json_fn = 'abc.ndjson'
data = json.load(open(json_fn, 'rb'))
pprint.pprint(data, depth=2)

but this just crashes with

Process finished with exit code 137 (interrupted by signal 9: SIGKILL)

I also tried:

import ijson

with open(json_fn) as f:
    items = ijson.items(f, 'item', multiple_values=True)  # "multiple values" needed as it crashes otherwise with a "trailing garbage parse error" (https://stackoverflow.com/questions/59346164/ijson-fails-with-trailing-garbage-parse-error)
    print('Data loaded - no processing ...')
    print("---items---")
    print(items)
    for item in items:
        print("---item---")
        print(item)

But this just returns:

Data loaded, now importing
---items---
<_yajl2.items object at 0x7f436de97440>

Process finished with exit code 0

The ndjson file contains valid ascii characters (as inspected with vi) but very long lines and is therefore not really comprehensible from a text editor.

The file starts like:

{"visitId":257057,"staticFeatures":[{"type":"CODES","value":"9910,51881,42833,486,4280,42731,2384,V5861,9847,3962,49320,3558,2720,4019,99092"},{"type":"visitID","value":"357057"},{"type":"VISITOR_ID","value":"68824"}, {"type":"ADMISSION_ID","value":"788457"},{"type":"AGE","value":"34"}, ...

What am I doing wrong and how can I process this file?

Upvotes: 1

Views: 742

Answers (1)

deadshot
deadshot

Reputation: 9061

You are using prefix item. For this to work json should have list as a top level element.

For example see below json

data2.json

[
    {
      "Identifier": "21979c09fc4e6574"
    },
    {
      "Identifier": "e6235cce58ec8b9c"
    }
 ]

Code:

with open('data2.json') as fp:
    items = ijson.items(fp, 'item')
    for x in items:
        print(x)

Output:

{'Identifier': '21979c09fc4e6574'}
{'Identifier': 'e6235cce58ec8b9c'}

Another Example

data.json

{
  "earth": {
    "europe": [
      {"name": "Paris", "type": "city", "info": {  }},
      {"name": "Thames", "type": "river", "info": {  }}
    ],
    "america": [
      {"name": "Texas", "type": "state", "info": {  }}
    ]
  }
}

Above json doesn't have list as top level element so I should provide the valid prefix to the ijson.items(). prefix should be 'earth.europe.item'

Code:

with open('data.json') as fp:
    items = ijson.items(fp, 'earth.europe.item')
    for x in items:
        print(x)

Output:

{'name': 'Paris', 'type': 'city', 'info': {}}
{'name': 'Thames', 'type': 'river', 'info': {}}

Upvotes: 1

Related Questions