CODEWITHSUNDEEP
javascript
user1265125
user1265125

Reputation: 2656

Parsing a large JSON array in Javascript

I'm supposed to parse a very large JSON array in Javascipt. It looks like:

mydata = [
    {'a':5, 'b':7, ... },
    {'a':2, 'b':3, ... },
    .
    .
    .
    ]

Now the thing is, if I pass this entire object to my parsing function parseJSON(), then of course it works, but it blocks the tab's process for 30-40 seconds (in case of an array with 160000 objects).

During this entire process of requesting this JSON from a server and parsing it, I'm displaying a 'loading' gif to the user. Of course, after I call the parse function, the gif freezes too, leading to bad user experience. I guess there's no way to get around this time, is there a way to somehow (at least) keep the loading gif from freezing?

Something like calling parseJSON() on chunks of my JSON every few milliseconds? I'm unable to implement that though being a noob in javascript.

Thanks a lot, I'd really appreciate if you could help me out here.

Upvotes: 0

Views: 1456

Answers (2)

nepeo
nepeo

Reputation: 509

I have never encountered a complete page lock down of 30-40 seconds, I'm almost impressed! Restructuring your data to be much smaller or splitting it into many files on the server side is the real answer. Do you actually need every little byte of the data?

Alternatively if you can't change the file @Cyrill_DD's answer of a worker thread will be able to able parse data for you and send it to your primary JS. This is not a perfect fix as you would guess though. Passing data between the 2 threads requires the information to be serialised and reinterpreted, so you could find a significant slow down when the data is passed between the threads and be back to square one again if you try to pass all the data across at once. Building a query system into your worker thread for requesting chunks of the data when you need them and using the message callback will prevent slow down from parsing on the main thread and allow you complete access to the data without loading it all into your main context.

I should add that worker threads are relatively new, main browser support is good but mobile is terrible... just a heads up!

Upvotes: 1

Cyril Duchon-Doris
Cyril Duchon-Doris

Reputation: 13949

You might want to check this link. It's about multithreading.

Basically :

var url = 'http://bigcontentprovider.com/hugejsonfile';

var f = '(function() {
            send = function(e) { 
                postMessage(e); 
                self.close();
            };
            importScripts("' + url + '?format=json&callback=send");
         })();';

var _blob = new Blob([f], { type: 'text/javascript' });

_worker = new Worker(window.URL.createObjectURL(_blob));
_worker.onmessage = function(e) { 
    //Do what you want with your JSON 
}
_worker.postMessage();

Haven't tried it myself to be honest...

EDIT about portability: Sebastien D. posted a comment with a link to mdn. I just added a ref to the compatibility section id.

Upvotes: 1

Related Questions