CODEWITHSUNDEEP
node.jsmongodbcsv
SILearner
SILearner

Reputation: 33

import multiple csv to mongodb

I know how to import a CSV file as a collection using mongoimport in shell/cmd prompt, or using GUI such as 3T studio. each line is a document with headerline as their title.

For example a csv file name Data.csv headerline and content are as follow:

Example

It can be done very easily using the above method if it is for just one file.

May I know is there a way to import multiple files (few hundred) where it can. -Separate each file as one collection -Use file name as collection name -Import all contents as each documents under the collections

Best is can use nodejs, but other method are more than welcome.

Thank you.

update

for i in `ls ~/te/*.csv`;  do 
   ./mongoimport -d test $i --type=csv --headerline  ; 
done

const exec = require('child_process').exec;
var yourscript = exec('bash mongoin.sh /te',
    (error, stdout, stderr) => {
        console.log(`${stdout}`);
        console.log(`${stderr}`);
        if (error !== null) {
            console.log(`exec error: ${error}`);
        }
    });

Upvotes: 3

Views: 3467

Answers (2)

Milad Resketi
Milad Resketi

Reputation: 108

This is python version of it.

import os
import subprocess

# directory of files
dir_files = 'C:\data'
# create list of all files
_, _, fns = next(os.walk(dir_files))
files = [os.path.join(dir_files, fn) for fn in fns]
# mongotool address
mongotool = r'C:\Program Files\MongoDB\Server\4.4\bin\mongoimport.exe'
# name of mongodb database
mydatabase = 'mydatabase'
# name of mongodb collection
mycollection = 'mycollection'
# import all files to mongodb
for fl in files:
    commands =[mongotool, '--db', mydatabase,
               '--collection', mycollection,
               '--file', fl,
               '--type', 'tsv',
               '--headerline']
    subprocess.Popen(commands, shell=True)

Upvotes: 1

Nicolas Modrzyk
Nicolas Modrzyk

Reputation: 14197

In bash:

for i in `ls ~/Desktop/*.csv`;  do 
   ./mongoimport -d local $i --type=csv --headerline  ; 
done

In batch:

forfiles /p c:\te /m *.csv /c "cmd /c mongoimport -d local @file --type csv --headerline"

Where 

 ~/Desktop/*.csv is the folder where the csv files are located.

And local is the mongodb database to import to.

The name of the collection will be picked up by the base filename of each csv file.

Put the content of the bash code in a script, say mymongo.sh:

for i in `ls $1`;  do 
   ./mongoimport -d local $i --type=csv --headerline  ; 
done

And then call it from node, with the following code:

const exec = require('child_process').exec;
var yourscript = exec('bash mymongo.sh /Users/niko/Desktop',
        (error, stdout, stderr) => {
            console.log(`${stdout}`);
            console.log(`${stderr}`);
            if (error !== null) {
                console.log(`exec error: ${error}`);
}});

or on windows, with the code in mymongo.bat

const exec = require('child_process').exec;
var yourscript = exec('cmd /c c:/te/mymongo.bat',
        (error, stdout, stderr) => {
            console.log(`${stdout}`);
            console.log(`${stderr}`);
            if (error !== null) {
                console.log(`exec error: ${error}`);
}});

Upvotes: 3

Related Questions