CODEWITHSUNDEEP
jsonmongodbmongoimport
user850234
user850234

Reputation: 3463

Import more than 1 json file using mongoimport

I am new to mongodb and want to know about importing a json file from one server to another. I tried the following command mongoimport -d test -c bik check.json and it works fine for me. Now i want to know when there are multiple json files how do i import all of them at a single go. I could not find any related document where it is written this is not possible. Please help me is this possible and how

Upvotes: 34

Views: 41029

Answers (13)

Teeks
Teeks

Reputation: 43

I used the solutions here to add a shell function to my bash profile for doing this quickly.

My example depends on the mongo export outputting each collection as a file with the collection name and .metadata.json extension.

function mimport() {
  for filename in *; do
    collection="${filename%.metadata.json}";
    mongoimport --db $1 --collection $collection --file $filename;
  done
}

Use in the path of the export files, passing the DB name to the command...

mimport my_db

Will load all collections into the DB at localhost.

Upvotes: 1

Tarida George
Tarida George

Reputation: 1303

Python:

from pathlib import Path
import subprocess

jsons_folder = "./out/"
mongodb_host = "172.22.0.3"
mongodb_port = "27017"
mongodb_user = "root"
mongodb_password = "1234"

for f in Path(jsons_folder).glob("*.json"):
    cmd = [
        "mongoimport", "-h", mongodb_host, "-p", mongodb_port, "--authenticationDatabase", "admin",
        "-u", mongodb_user, "-p", mongodb_password,
        "--db", "callscoring", "--collection", "scoring_result_entry", str(f.absolute())
    ]
    subprocess.run(cmd)

Upvotes: 0

Frederick Zhang
Frederick Zhang

Reputation: 3683

Not sure whether it's a new feature, but mongoimport now can actually read from stdin. So what one can do to import multiple JSON files is as simple as

cat *.json | mongoimport --uri "mongdb://user:password@host/db?option=value" --collection example

I'm using mongodb-tools v4.2.0 btw.

UPDATE

mongodbimport can potentially consume a high amount of memory which may cause the program to be kill by system OOM. My machine's got 32GB RAM and this happened consistently when I tried to import ~10GB of data which is stored in RAM disk.

To divide a relatively large job into batches:

#!/usr/bin/env bash

declare -a json_files=()
for f in *.json; do
    json_files+="$f"
    if [[ "${#json_files[@]}" -ge 1000 ]]; then
        cat "${json_files[@]}" | mongoimport --uri="mongodb://user:pass@host/db" --collection=examples -j8 #--mode=upsert --upsertFields=id1
        json_files=()
    fi
done

Upvotes: 3

user854301
user854301

Reputation: 5503

One line solution:

for /F %i in ('dir /b c:\files\*.json') do mongoimport.exe /d db /c files /file c:\file\%i

Upvotes: 2

Juan Miguel Díaz Pérez
Juan Miguel Díaz Pérez

Reputation: 171

Linux:

> cat one.json two.json > three.json

> mongoimport --db foo --collection baz --file three.json"

Or, all files in the folder :

> cat *.json > big.json

> mongoimport --db foo --collection baz --file "big.json"

Upvotes: 2

Sam Kenny
Sam Kenny

Reputation: 395

Another one line solution (assuming you are in the folder where the json files are):

ls | sed 's/.json$//' | xargs -I{} mongoimport -d DATABASE_NAME -c {} {}.json

Upvotes: 2

Luan Rodrigues
Luan Rodrigues

Reputation: 11

I'm going to show how to import many collections efficiently using only the Linux's terminal (it also works in Mac).

You must have all json files at the same folder and the file's name should be the collection that will be imported to your database.

So, let's begin, open the folder that contains your json files. Replace the <DATABASE> to your database name, then execute the line below:

for collection in $(ls | cut -d'.' -f1); do mongoimport --db <DATABASE> --collection ${collection} --file ${collection}.json; done

But what is going on there?

First of all, you have to keep in mind that the parentheses will be executed first. In this case, it creates a list of all files getting just the name of each file (removing it's extension).

Secondly, all list will be added to a loop "for" in a local variable called collection (this variable's name could be anything you want)

Thirdly, the "do" execute the import line(*)

Finally the "done", finish the loop.

(*) The import line is composed by "mongoimport" that requires the database name "--db", the collection name "--collection", and the file name "--file". These requirements has been filled by the variable "$collection" created on the "for" stuff

I hope helped someone! Good luck guys :)

Upvotes: 1

helpdoc
helpdoc

Reputation: 1990

You can do it by this way also :

for filename in *; do mongoimport --db <Database> --collection <Collection Name> --file $filename; done

Upvotes: 15

Eduardo
Eduardo

Reputation: 85

For windows bat file. This would be way better if you have a list of json files in the folder. and the collection name matches the name in files 

@echo off
for %%f in (*.json) do (
    "mongoimport.exe" --db databasename --collection %%~nf --drop --file %%f
)
pause

Upvotes: 4

Kanagavelu Sugumar
Kanagavelu Sugumar

Reputation: 19260

This worked for me in MAC OS X 

find . -regex '.*/[^/]*.json' | xargs -L 1 mongoimport --db DB_NAME -u USER_NAME -p PASSWORD --collection COLLECTION_NAME  --file

Upvotes: 6

romaninsh
romaninsh

Reputation: 10664

I came up with a more elegant way to automatically import ALL collections:

ls -1 *.json | sed 's/.json$//' | while read col; do 
    mongoimport -d db_name -c $col < $col.json; 
done

I hope this is helpful.

Upvotes: 39

Tomi Kokkonen
Tomi Kokkonen

Reputation: 428

Windows Batch version:

@echo off
for %%f in (*.json) do (
    "mongoimport.exe" --jsonArray --db databasename --collection collectioname --file %%~nf.json
)

Upvotes: 17

Sergio Tulentsev
Sergio Tulentsev

Reputation: 230471

You can always write some shell scripts.

colls=( mycoll1 mycoll2 mycoll5 )

for c in ${colls[@]}
do
  mongoimport -d mydb -c $c.json
done

Upvotes: 29

Related Questions