CODEWITHSUNDEEP
javascriptmysqlnode.jsperformancenpm
mixalbl4
mixalbl4

Reputation: 3995

The fastest way to work with a large array of data in node

I have some db with 4 000 000 numbers and I need to check if some number is exist in db

Database exapmle: [177,219,245,309,348,436,...]

I tried use mysql table but it takes 1300ms for check if some 1 number is exist

It very long time and I`ve check some another ways:

1) Push all numbers to array and check it with indexOf

2) Push all numbers to string and check it with indexOf

3) Push all numbers to object and check it with hasOwnProperty

4) Push all numbers to object and check value

I`ve made 100 checks in all of this ways and measured the time:

try [...].indexOf(...) === 0
Total: 726ms

try String(...).indexOf(...) === 0
Total: 1915ms

try {...}.hasOwnProperty(...) === true
Total: 4ms

try {...}[...] === true === true
Total: 4ms

The code of tests:

// Reads db json with ~4 000 000 elements
var arr = utils.readJSON('members.json');
// Converts db to string type
var arr_string = arr.toString();
// Makes empty object
var arr_obj = {};
// Converts db to object type
for(var i in arr){
    arr_obj[arr[i]] = true;
}
// Makes empty array
var checkList = [];
// Gets 100 different values from different places of db
for(var i = 0; i < 100; i++){
    checkList.push(arr[Math.floor(arr.length / 100 * i)]);
}
// Main time variable
var time;

// test
time = new Date().getTime();
for (var i in checkList) {
    console.log('try [...].indexOf('+ checkList[i]+') === ' + arr.indexOf(checkList[i]));
}
console.log('Total: ' + (new Date().getTime() - time) + 'ms');

// test
time = new Date().getTime();
for (var i in checkList) {
    console.log('try String(...).indexOf(' + checkList[i] + ') === ' + arr_string.indexOf(checkList[i]));
}
console.log('Total: ' + (new Date().getTime() - time) + 'ms');

// test
time = new Date().getTime();
for (var i in checkList) {
    console.log('try {...}.hasOwnProperty(' + checkList[i] + ') === ' + arr_obj.hasOwnProperty(checkList[i]));
}
console.log('Total: ' + (new Date().getTime() - time) + 'ms');

// test
time = new Date().getTime();
for (var i in checkList) {
    console.log('try {...}[' + checkList[i] + '] === true === ' + (arr_obj[checkList[i]] === true));
}
console.log('Total: ' + (new Date().getTime() - time) + 'ms');

I believe that the best way to perform this task is Object with hasOwnProperty, but what different ways do you know?

Maybe I should try using other databases?

Upvotes: 0

Views: 1614

Answers (1)

rsp
rsp

Reputation: 111506

It started out as a comments but it didn't fit so I post it as an answer. As other has already pointed out, use an index on the column that you want to search or sort anything by. Always. Not only in this example.

If it isn't enough then consider using some other data store like Redis that may be more efficient to handle your particular use case.

Upvotes: 3

Related Questions