CODEWITHSUNDEEP
node.jsmongodbcsv
MonsterWimp757
MonsterWimp757

Reputation: 1217

Populate MongoDB from CSV using NodeJS

I am trying to populate my MongoDB using data from a CSV file. There are currently no databases or collections in my MongoDB and I would like to create these using an update function that creates objects parsed from a csv file.

I am using ya-csv to parse my csv file and the mongodb driver for node.

My code looks like this:

var csv = require('ya-csv');
var fs = require('fs');
var MongoClient = require('mongodb').MongoClient;
var Server = require('mongodb').Server;
var mongoclient = new MongoClient(new Server('localhost', 27017, {'native_parser' : true}));

var reader = csv.createCsvFileReader('YT5.csv', {columnsFromHeader:true,'separator':   ','});
reader.addListener('data', function(data){
var nameHolder = data.name;
//I have no issue getting the needed variables from my csv file
mongoclient.db(nameHolder).collection('assets').update({assetId:data.assetId,name:data.name},{upsert:true},function(err,updated){if(err){console.log(err)}});

reader.addListener('end', function(data){
console.log("done");
}

I have not created the databases or collections for this, but can it do this for me with this update? I get an error:

[Error: Connection was destroyed by application]

When I run this, the databases get created but they're empty. Any help would be greatly appreciated.

Upvotes: 3

Views: 7599

Answers (2)

anapsix
anapsix

Reputation: 2085

Unless there is a specific need to use NodeJS, say to not only reorder but make some complex modification to fields read from CSV file, use mongoimport.

If all you need to do is to skip and reorder some fields - work the fields with simple awk, skipping and changing order as needed:

cat /tmp/csv | awk -F',' 'BEGIN{OFS=","} {print $1,$2,$4,$3,$10}' | mongoimport --type csv --db test --collection csv_import

Additionally, if there is a need to change collection or db name based on the csv values (field 10 in this example is used as db name and field 11 as collection):

cat /tmp/csv | awk -F',' 'BEGIN{OFS=","} {print $1,$2,$4,$3,$10 | "mongoimport --type csv --db "$10" --collection "$11 }'

Or you could split the file into the per db/collection chunks first.

Upvotes: 6

Giovanni Bitliner
Giovanni Bitliner

Reputation: 2072

If you convert the csv rows in a javascript array (each row is an object), you can use https://github.com/bitliner/MongoDbPopulator

Upvotes: 0

Related Questions