CODEWITHSUNDEEP
mongodbmeteor
gabrielhpugliese
gabrielhpugliese

Reputation: 2588

How should I insert a bunch of data in Meteor Mongodb?

It's taking a long time to save 500+ Facebook friends in MongoDB and I think I'm doing it so wrong. I'll paste how I'm doing the insertion:

models.js:

Friends = new Meteor.Collection('friends');
Friend = {
set : function(owner, friend) {
    var user_id = get_user_by_uid(friend['uid']);
    return Friends.update({uid: friend['uid'], owner: owner}, {$set:{
        name : friend['name'],
        pic_square : 'https://graph.facebook.com/'+friend['uid']+'/picture?width=150&height=150',
        pic_cover : friend['pic_cover'],
        uid : friend['uid'],
        likes_count : friend['likes_count'],
        friend_count : friend['friend_count'],
        wall_count : friend['wall_count'],
        age : get_age(friend['birthday_date']),
        mutual_friend_count : friend['mutual_friend_count'],
        owner : owner,
        user_id : user_id ? user_id['_id'] : undefined
    }}, {upsert: true});
}
}

server.js:

// First get facebook list of friends   
friends = friends['data']['data'];

_.each(friends, function(friend){
    Friend.set(user_id, friend);
});

The loads go high with 2+ users and it takes ages to insert on the database. What should I change here ?

Upvotes: 0

Views: 710

Answers (1)

DoctorPangloss
DoctorPangloss

Reputation: 3073

The performance is bad for two reasons I think.

First, you are experiencing minimongo performance, not mongodb performance, on the client. minimongo can't index, so upsert is expensive—it is O(n^2) expensive on database size. Simply add a if (Meteor.isSimulation) return; statement into your model just before the database updating.

Take a look at some sample code to see how to organize your code a bit, because Friend.set(user_id, friend) should be occurring in a method call, conventionally, defined in your model.js. Then, it should escape if it is detected as a client simulating the call, as opposed to the server executing it.

Second, you are using uid and owner like a key without making them a key. On your server startup code, add Friends._ensureIndex({uid:1, owner:1}).

If none of this works, then your queries to Facebook might be rate-limited in some way.

Check out https://stackoverflow.com/a/8805436/1757994 for some discussion about the error message you'd receive if you are being rate limited.

They almost certainly do not want you copying the graph the way you are doing. You might want to consider not copying the graph at all and only getting the data on a use-basis, because it very rapidly becomes stale anyway.

Upvotes: 4

Related Questions