CODEWITHSUNDEEP
pythonjsonmemorydumpsimplejson
Charlie Epps
Charlie Epps

Reputation: 3675

having problems when dumps dict to json?

Here, I want to dump a "big" dict into json, as following:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import simplejson as json

doc = {}
# appending the doc, so that the doc is more than 2G
.....

json_doc = json.dumps(doc)

Then i got the following error message:

  File "C:\Python27\lib\site-packages\simplejson\__init__.py", line 286, in dump
s
    return _default_encoder.encode(obj)
  File "C:\Python27\lib\site-packages\simplejson\encoder.py", line 228, in encod
e
    chunks = list(chunks)
MemoryError

How can I fix it? thanks!

Upvotes: 1

Views: 2353

Answers (1)

jfs
jfs

Reputation: 414179

If there is barely enough memory you could try to encode the object to json incrementally:

import json
import sys

d = dict.fromkeys(range(10))
for chunk in json.JSONEncoder().iterencode(d):
    print(chunk) # print each chunk on a newline for demonstration

Don't accumulate the output in a string use file/socket and write/send chunks immediately.

Output

{
"0"
: 
null
, 
"1"
: 
null
, 
"2"
: 
null
, 
"3"
: 
null
, 
"4"
: 
null
, 
"5"
: 
null
, 
"6"
: 
null
, 
"7"
: 
null
, 
"8"
: 
null
, 
"9"
: 
null
}

Upvotes: 5

Related Questions