CODEWITHSUNDEEP
pythonujson
Haminator
Haminator

Reputation: 11

How to fix memory error caused by ujson.loads

I am trying to load a 2GB json file using ujson.loads, and getting a memory error:

  File "dict.py", line 70, in <module>
    dct = load_dict_file()
  File "dict_load.py", line 25, in load_latest_dict
    dct = ujson.loads(dict_txt)
MemoryError

I'm running my code on an EC2 machine (Linux) with enough free memory, using Python 3.7:

[ec2-user@ip]$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15463         269       14330          11         863       14911
Swap:             0           0           0

the code runs as expected on my local machine (windows), so not sure what seems to cause the memory error. I considered using ijson, but it looks like it doesn't suit my use case (pasring the json file to load it key-by-key to rows in a table, using sqlalchemy session).

Thanks

Upvotes: 0

Views: 347

Answers (0)

Related Questions