CODEWITHSUNDEEP
node.jsmongodbmongoosemigration
codephobia
codephobia

Reputation: 1590

migrating data from one mongodb database to another with mongoose

I have to migrate about 19k accounts from one database to another, and change their formatting, etc. Wondering if anyone has a solid method for doing this, perhaps with a simple node.js script. I have searched for an example online, but haven't been able to find anything.

Upvotes: 2

Views: 1662

Answers (2)

cangokceaslan
cangokceaslan

Reputation: 482

I implemented a npm package for this purpose. Using my package, you can give different sets of configurations and queries to move datas from one cluster/model into many cluster/models

configurations:

model:       // preferred model name
schema:      // Schema for the collection - reading the data
query:      // Query for searching over the collection
populate:   // Relation with other collections - Same with .populate() method
skip:       // skipping N values in the collection - same with .skip() method
sort:       // sorting - same with .sort() method
select:     // selecting keys - same with .select()
concurrent: // concurrent adjusts the pagination fetching constant
limit:      // total data will be fetched - same with .limit() method

example:

const MongooseTruck = require("mongoose-truck")
const conn1 = await MongooseTruck.connect("database_connection_string"),
    conn2 = await MongooseTruck.connect("database_connection_string"),
    conn3 = await MongooseTruck.connect("database_connection_string");
const parsedData = await MongooseTruck.create(conn1, [conn2, conn3]).loadFrom({
    model: 'People',
    schema: PeopleSchema,
    query: {
        name: "John"
    },
    populate: {
        path: 'stars'
    },
    skip: 0,
    sort: {
        _id: -1
    },
    select: "_id name surname age",
    concurrent: 20,
    limit: 100
}).emptyTo({
    model: 'People',
    schema: PeopleSchema
}).start({
    parse: true
})

GitHub: mongoose-truck

Npm: mongoose-truck

Upvotes: 0

Zlatko
Zlatko

Reputation: 19588

I would simply stream this with something simple like monk.js. Here's an example program (no shell here so it might not be 100% accurate):

var monk = require('monk');
var src = monk('localhost/my-database');
var dest = monk('remote-host/remote-database');

var users = src.get('users');
var remoteUsers = dest.get('newUsers');

users.find({}, { stream: true })
.each(function(user){

    // change some data
    user.profileImage = user.image;
    delete user.legacyPropery;
    user.name = user.firstName + ' ' + user.lastName;
    return user;
})
.error(function(err){})
.success(function(){});

That's the gist of it. Even if you have multiple related collections that you want to migrate as well (like, coments or stuff), streaming and moving one by one sounds like the best option. You can run like this in parallel (several processes or use this module in cluster module several times), you can count success/errors etc.

Almost the same as if you'd mongodump and then mongorestore on the new database, then run a converter script there. (with an $update or something.).

Upvotes: 2

Related Questions