CODEWITHSUNDEEP
javascriptgoogle-chrome-extensionindexeddb
surya
surya

Reputation: 1359

Inserting large quantities in IndexedDB's objectstore blocks UI

I want to save some ~35000 objects in my IndexedDB's objectstore. I am using below code to insert.

AddListings = function (x2j_list_new, callback) {   
    var transaction = db.transaction(["listings"], IDBTransaction.READ_WRITE);
    var count = 0;
    transaction.oncomplete = function (event) {
        if (callback) {
            console.log('x2jShowListing Added ' + count + '/' + x2j_list_new.length);
                callback([count, x2j_list_new.length]);
            }
    };
    transaction.onerror = function (e) {
       console.log("myError: ", e);  
       if (callback) {
          callback(false);
       }
    };
    var store = transaction.objectStore("listings");

    $.each(x2j_list_new, function (index0, item0) {
        var request = store.put(item0);
        request.onsuccess = function (event) {
            count++;
            // event.target.result  
            };
        });
    });        
};

The above code works fine, but looping and inserting over ~35000 objects makes the UI unresponsive for ~200 seconds. I thought maybe i can use WebWorkers, but IndexedDB is not available inside WebWorkers. I tried to find a way to bulk insert, couldn't find one. Any ideas of how to insert large quantities of objects without blocking the UI?

Upvotes: 21

Views: 14840

Answers (5)

Alejadro Xalabarder
Alejadro Xalabarder

Reputation: 1630

do it all in one transaction, for example

    function populateData (db, storetable, records, whencomplete, whenfails) 
    {
        var transaction = db.transaction (storetable, 'readwrite');
        var objectStore = transaction.objectStore (storetable);

        for (var ii = 0; ii < records.length ; ii ++) 
            objectStore.put (records[ii]);

        if (whencomplete)
            transaction.oncomplete = function (event) { whencomplete (event);  }

        if (whenfails)
            transaction.onerror = function (event) { whenfails (event);  }
    }

see also function populateData in the example https://github.com/mdn/dom-examples/blob/main/indexeddb-examples/idbcursor/scripts/main.js

Upvotes: 0

buley
buley

Reputation: 29208

You're doing everything right by using callbacks.

The Webworker API has not yet been implemented by any major browser. Interestingly, it is expected to be synchronous. The regular API is async for the exact reason you describe -- it's not supposed to block the UI thread.

Using callbacks is the way to avoid lock ups, but at 35k objects you're clearly seeing this paradigm break down. Unfortunately, IDB performance is not yet on par with WebSQL from the benchmarks I've seen.

With Chrome's LevelDB there's been some new experimental backends (FF is SQLite ) but I think your experience proves that there's some room for improvement.

Upvotes: 0

Doug Reeder
Doug Reeder

Reputation: 739

You're on the right track, but you're asking the browser to store 35,000 objects before it's had a chance to finish storing one. Here's code which asynchronously waits for one request to finish before starting the next (but using the same transaction):

    openRequest = window.indexedDB.open("MyDatabase", 1);
    openRequest.onerror = function(event) {
        console.error(event);
    };
    openRequest.onsuccess = function (event) {
        var db = openRequest.result;
        db.onerror = function(event) {
            // Generic error handler for all errors targeted at this database's requests
            console.error(event.target);
            window.alert("Database error: " + event.target.wePutrrorMessage || event.target.error.name || event.target.error || event.target.errorCode);
        };
        var transaction = db.transaction('item', "readwrite");
        var itemStore = transaction.objectStore("item");
        putNext();

        function putNext() {
            if (i<items.length) {
                itemStore.put(items[i]).onsuccess = putNext;
                ++i;
            } else {   // complete
                console.log('populate complete');
                callback();
            }
        }           
    };

Upvotes: 42

surya
surya

Reputation: 1359

I am splitting the array in chunks of 500 and using setTimeout instead of for loop. Now the UI responds little better than before

Upvotes: 0

Marius Kjeldahl
Marius Kjeldahl

Reputation: 6824

Wild guess from my side, but if WebSQL is available from what is known as a "background page", and assuming the bandwidth of messaging between the front- and backpage does not lock up the UI in the same way, maybe a background page with intra page message could be utilized?

Upvotes: 0

Related Questions