Reputation: 1590
I have to migrate about 19k accounts from one database to another, and change their formatting, etc. Wondering if anyone has a solid method for doing this, perhaps with a simple node.js script. I have searched for an example online, but haven't been able to find anything.
Upvotes: 2
Views: 1662
Reputation: 482
I implemented a npm package for this purpose. Using my package, you can give different sets of configurations and queries to move datas from one cluster/model into many cluster/models
configurations:
model: // preferred model name
schema: // Schema for the collection - reading the data
query: // Query for searching over the collection
populate: // Relation with other collections - Same with .populate() method
skip: // skipping N values in the collection - same with .skip() method
sort: // sorting - same with .sort() method
select: // selecting keys - same with .select()
concurrent: // concurrent adjusts the pagination fetching constant
limit: // total data will be fetched - same with .limit() method
example:
const MongooseTruck = require("mongoose-truck")
const conn1 = await MongooseTruck.connect("database_connection_string"),
conn2 = await MongooseTruck.connect("database_connection_string"),
conn3 = await MongooseTruck.connect("database_connection_string");
const parsedData = await MongooseTruck.create(conn1, [conn2, conn3]).loadFrom({
model: 'People',
schema: PeopleSchema,
query: {
name: "John"
},
populate: {
path: 'stars'
},
skip: 0,
sort: {
_id: -1
},
select: "_id name surname age",
concurrent: 20,
limit: 100
}).emptyTo({
model: 'People',
schema: PeopleSchema
}).start({
parse: true
})
GitHub: mongoose-truck
Npm: mongoose-truck
Upvotes: 0
Reputation: 19588
I would simply stream this with something simple like monk.js. Here's an example program (no shell here so it might not be 100% accurate):
var monk = require('monk');
var src = monk('localhost/my-database');
var dest = monk('remote-host/remote-database');
var users = src.get('users');
var remoteUsers = dest.get('newUsers');
users.find({}, { stream: true })
.each(function(user){
// change some data
user.profileImage = user.image;
delete user.legacyPropery;
user.name = user.firstName + ' ' + user.lastName;
return user;
})
.error(function(err){})
.success(function(){});
That's the gist of it. Even if you have multiple related collections that you want to migrate as well (like, coments or stuff), streaming and moving one by one sounds like the best option. You can run like this in parallel (several processes or use this module in cluster module several times), you can count success/errors etc.
Almost the same as if you'd mongodump
and then mongorestore
on the new database, then run a converter script there. (with an $update or something.).
Upvotes: 2