CODEWITHSUNDEEP
node.jsmongodbcsv
EraseMe
EraseMe

Reputation: 1

CSV, Download or manipulate right away for mongoDB in node.js

I am working on a project and in which i need to manipulate a CSV file hosted on a distant URL and then send the formatted content to a distant mongodb server. I have tried many approaches without much success so far, so i was wondering what you would suggest ? Is it easier to download the distant file locally and then reupload it or to just load the contents in memory and insert it right away?

Thanks

Upvotes: 0

Views: 177

Answers (1)

Michał Karpacki
Michał Karpacki

Reputation: 2658

I assume you want to save every entry from the CSV to a mongo

An easy way with scramjet:

const {StringStream} = require('scramjet');
const request = require('request');
const mongo = require('mongo-db');

const db = new mongodb(url, cfg.collection, true);

StringStream.from(request.get('https://example.com/csvs/first.csv')) 
                                    // get your CSV as a stream and pass it to scramjet
    .CSVParse()                     // parse the stream as CSV
    .setOptions({maxParallel: 32})  // set maximum parallel operations
    .map(x => {                     // map every object
        const y = await manipulate(x);
        return JSON.parse(y);
    })
    .consume((y) => db.insert(y));  // insert into database

Disclamer: I'm the author of scramjet.

Upvotes: 2

Related Questions