CODEWITHSUNDEEP
node.jsangularjsmongodbcsv
Gary
Gary

Reputation: 11

how insert csv data to mongodb with nodejs

Hi im developing an app with nodeJS, express and a mongoDB, i need to take users data from a csv file and upload it to my database this db has a schema designed with mongoose.

but i don know how to do this, what is the best approach to read the csv file check for duplicates against the db and if the user (one column in the csv) is not here insert it?

are there some module to do this? or i need to build it from scratch? im pretty new to nodeJS i need a few advices here

Thanks

this app have an angular frontend so the user can upload the file, maybe i should read the csv in the front end and transform it into an array for node, then insert it?

Upvotes: 1

Views: 7801

Answers (1)

Jake Sellers
Jake Sellers

Reputation: 2439

Use one of the several node.js csv libraries like this one, and then you can probably just run an upsert on the user name.

An upsert is an update query with the upsert flag set to true: {upsert: true}. This will insert a new record only if the search returns zero results. So you query may look something like this:

 db.collection.update({username: userName}, newDocumentObj, {upsert: true})

Where userName is the current username you're working with and newDocumentObj is the json document that may need to be inserted.

However, if the query does return a result, it performs an update on those records.

EDIT: I've decided that an upsert is not appropriate for this but I'm going to leave the description.

You're probably going to need to do two queries here, a find and a conditional insert. For this find query I'd use the toArray() function (instead of a stream) since you are expecting 0 or 1 results. Check if you got a result on the username and if not insert the data.

Read about node's mongodb library here.

EDIT in response to your comment: It looks like you're reading data from a local csv file, so you should be able to structure you program like:

function connect(callback) {
    connStr = 'mongodb://' + host + ':' + port + '/' + schema;  //command line args, may or may not be needed, hard code if not I guess
    MongoClient.connect(connStr, function(err, db) {
        if(err) {
            callback(err, null);
        } else {
            colObj = db.collection(collection); //command line arg, hard code if not needed
            callback(null, colObj);
        }
    });
}

connect(function(err, colObj) {
    if(err) {
        console.log('Error:', err.stack);
        process.exit(0);
    } else {
        console.log('Connected');
        doWork(colObj, function(err) {
            if(err) {
                console.log(err.stack);
                process.exit(0);
            }
        });
    }
});


function doWork(colObj, callback) {
    csv().from('/path/to/file.csv').on('data', function(data) {
        //mongo query(colObj.find) for data.username or however the data is structured
        //inside callback for colObj.find, check for results, if no results insert data with colObj.insert, callback for doWork inside callback for insert or else of find query check
    });
}

Upvotes: 5

Related Questions