CODEWITHSUNDEEP
pythonarraysjsonpython-3.xdeserialization
Sreedharan
Sreedharan

Reputation: 83

Convert array of JSON String to array of object in Python

I having array of JSON string, I need to convert it as array of object (i.e., convert the JSON to respective object) without a for loop.

Source Code: (Input data)

data = ['[1,2,3]', '[4,5,6]', '[7,8,9]']

Required Output:

[[1,2,3], [4,5,6], [7,8,9]]

I already using the following solution

import json

data = ['[1,2,3]', '[4,5,6]', '[7,8,9]']
output = []
for item in data:
    output.append(json.loads(item))

Currently I'm having a very large number of JSON strings (approx 100K records) and moreover each JSON String array internally contains the record approx 50K. While on execution it takes more than 3GB of RAM for processing.

Note: Implicitly the output is a 2-dim array [][]. 1st dimension is approx 100K records 2nd dimension contains approx 50K records. Totally 100K * 50K items.

While on conversion it takes more time to convert the JSON (for the above approach). Kindly assist me the idea to convert the JSON string without a for loop.

Upvotes: 3

Views: 13082

Answers (2)

klim
klim

Reputation: 1269

ujson could make your code faster

import time
import json
import ujson

a_list = list(range(5000))
data = [str(a_list)] * 10000

s = time.time()

output = []
for item in data:
    output.append(json.loads(item))

print("json : %s" % (time.time()-s))

s = time.time()

output = []
for item in data:
    output.append(ujson.loads(item))

print("ujson : %s" % (time.time()-s))

On my PC...

json : 10.048374891281128
ujson : 6.533677577972412

Upvotes: 0

Anup Yadav
Anup Yadav

Reputation: 3005

Now solution looks wired but this works and will be useful for you in optimization. Convert the complete list into str then remove all ' single commas with str function and the apply json loads, hurray this has got worked for me. 

data = ['[1,2,3]', '[4,5,6]', '[7,8,9]']
r = str(data).replace("'",'')

import json
data = json.loads(r)

now your data will be of list of list without looping. You can achieve this.

[[1, 2, 3], [4, 5, 6], [7, 8, 9]]

Upvotes: 2

Related Questions