CODEWITHSUNDEEP
javascriptjsonangularjsindexeddbdexie
poptard
poptard

Reputation: 61

IndexedDB and large amount of inserts on Angular app

I'm struggling with amounts of 20-50k JSON object response from server which I should insert into our indexeddb datastore.

Response is repeated with foreach and every single row is added with each. Calls with response less than 10k rows are working fine and inserted within a minute or so. But when the amounts get larger, the database goes unresponsive after a while and returns this error message

"db Error err=transaction aborted for unknown reason"

I'm using a Dexie wrapper for the database and an angular wrapper for dexie called ngDexie. 

var deferred = $q.defer();
var progress = 0;
 // make the call
 $http({
       method: 'GET',
       headers: headers,
       url: '/Program.API/api/items/getitems/' + user
            }).success(function (response) {

        // parse response
        var items = angular.fromJson(response);
        // loop each item
        angular.forEach(items, function (item) {    
                    // insert into db
                    ngDexie.put('stuff', item).then(function () {
                        progress++;
                        $ionicLoading.show({
                            content: 'Loading',
                            animation: 'fade-in',
                            template: 'Inserting items to db: ' + progress
                                       + '/' + items.length,
                            showBackdrop: true,
                            maxWidth: 200,
                            showDelay: 0

                        });

                        if (progress == items.length) {
                            setTimeout(function () {
                                $ionicLoading.hide();
                            }, 500);
                            deferred.resolve(items);

                        }      
                    });
                });
            }).error(function (error) {
                $log('something went wrong');
                $ionicLoading.hide();
            });
            return deferred.promise;

Do I have the wrong approach with dealing with the whole data in one chunk? Could there be better alternatives? This whole procedure is only done once when the user opens up the site. All help is greatly appreciated. The target device is tablets running Android with Chrome.

Upvotes: 0

Views: 2402

Answers (3)

Francesco
Francesco

Reputation: 1148

I had problems with massive bulk insert (100.000 - 200.000 records). I've solved all my IndexedDB performance problems using bulkPut() from Dexie library. It has this important feature:

Dexie has a kick-ass performance. It's bulk methods take advantage of a not well known feature in indexedDB that makes it possible to store stuff without listening to every onsuccess event. This speeds up the performance to a maximum.

Dexie: https://github.com/dfahlander/Dexie.js

BulkPut() -> http://dexie.org/docs/Table/Table.bulkPut()

Upvotes: 0

Martin Diphoorn
Martin Diphoorn

Reputation: 81

If you insert many new records i would suggest using add. This was added for performance reasons. See the documentation here:

https://github.com/FlussoBV/NgDexie/wiki/ngDexie.add

Upvotes: 0

Kristof Degrave
Kristof Degrave

Reputation: 4180

Since you are getting a unknown error, there is something going wrong with I/O. My guess is the db underneath has troubles handling the amout of data. May try to split up in batches with a maximum of 10k each.

A transaction can fail for reasons not tied to a particular IDBRequest. For example due to IO errors when committing the transaction, or due to running into a quota limit where the implementation can't tie exceeding the quota to a partcular request. In this case the implementation MUST run the steps for aborting a transaction using the transaction as transaction and the appropriate error type as error. For example if quota was exceeded then QuotaExceededError should be used as error, and if an IO error happened, UnknownError should be used as error.

you can find this in the specs

An other possibility, do you have any indexes defined on the objectstore? Because for every index you have, that index needs to be maintained with every insert.

Upvotes: 1

Related Questions