CODEWITHSUNDEEP
javascriptjsonmongodb
Sampath Liyanage
Sampath Liyanage

Reputation: 4896

Import JSON with mongo script

I'm trying to write a mongo script to import a jsonArray from a JSON file. My script is in .js format and I execute it with load() command in mongo shell. Is it possible to do it with a mongo script?

I know I can use mongoimport instead. But I want to know a way to do it with a script.

The contents of my current script in which the import part is missing is given below..

var db = connect("localhost:27017/fypgui");
//Import json to "crimes" collection here 
var crimes = db.crimes.find();
while (crimes.hasNext()){
    var item = crimes.next();
    var year =(item.crime_date != null)?(new Date(item.crime_date)).getFullYear():null;
    db.crimes.update( {_id: item._id}, {$set: {crime_year: year}});
}

Upvotes: 7

Views: 12276

Answers (5)

Arno van Oordt
Arno van Oordt

Reputation: 3510

I was looking for this for quite a while and the given anwers were either not working or just providing a workaround.

Luckily I found the answer.. You can use fs.readFileSync to read (json) files. For completeness sake here is my entire script:

const dbName = process.env.DB_NAME || 'test_db';
const dbUser = process.env.DB_USER || 'test_user';
const dbPassword = process.env.DB_PASSWORD || 'secret';

db.createUser({
  user: dbUser,
  pwd: dbPassword,
  roles: [
    {
      role: 'readWrite',
      db: dbName,
    },
  ],
});

// Create and seed collections:

const collections = [
  'some_collection',
  'another_collection',
];

for (let i = 0; i < collections.length; i++) {
  const collectionName = collections[i];
  db.createCollection(collectionName);

  let jsonFile;
  try {
    jsonFile = fs.readFileSync(`/docker-entrypoint-initdb.d/mongo.${collectionName}.json`, 'utf8');
  }
  catch (err) {
    console.log(`NOTE: mongo.${collectionName}.json not found! Skip seeding of ${collectionName} collection..`)
    continue;
  }

  const json = JSON.parse(jsonFile);

  // Parse the $oid ids that Mongo uses in exports, but cant use in insertMany:
  for (let i = json.length; i--;) {
    json[i]._id = ObjectId(json[i]._id.$oid);
  }

  db[collectionName].insertMany(json);
}

Upvotes: 0

igor.sol
igor.sol

Reputation: 647

It is possible to get file content as text using undocumented cat() function from the mongo shell:

var text = cat(filename);

If you are interested cat() and other undocumented utilities such as writeFile are defined in this file: shell_utils_extended.cpp

Having file's content you can modify it as you wish or directly pass it to JSON.parse to get JavaScript object:

jsObj = JSON.parse(text);

But be careful: unfortunately JSON.parse is not an equivalent of mongoimport tool in the sense of its JSON parsing abilities.

mongoimport is able to parse Mongo's extended JSON in canonical format. (Canonical format files are created by bsondump and mongodump for example. For more info on JSON formats see MongoDB extended JSON).

JSON.parse does not support canonical JSON format. It will read canonical format input and will return JavaScript object but extended data type info present in canonical format JSON will be ignored.

Upvotes: 3

Amin Jellali
Amin Jellali

Reputation: 11

Hey I know this is not relevent but, every time I need to import some jsons to my mongo db I do some shity copy, paste and run, until I had enough!!! If you suffer the same I v written a tiny batch script that does that for me. Intrested?

https://github.com/aminjellali/batch/blob/master/mongoImporter.bat

@echo off
title = Mongo Data Base importing tool
goto :main

:import_collection
    echo importing %~2
    set file_name=%~2
    set removed_json=%file_name:.json=%
    mongoimport --db %~1 --collection %removed_json% --file %~2
goto :eof

:loop_over_files_in_current_dir
    for /f %%c in ('dir /b *.json') do call :import_collection %~1 %%c
goto :eof

:main
    IF [%1]==[] (
    ECHO FATAL ERROR: Please specify a data base name
    goto :eof
    ) ELSE (
    ECHO @author amin.jellali
    ECHO @email [email protected]
    echo starting import...
    call :loop_over_files_in_current_dir %~1
    echo import done...
    echo hope you enjoyed me
    )
goto :eof

Upvotes: 1

David H. Bennett
David H. Bennett

Reputation: 1842

There is another answer to this question. Even though it's a bit old I am going to respond.

It is possible to do this with the mongo shell.

You can convert your JSON to valid JavaScript by prefixing it with var myData= then use the load() command to load the JavaScript. After the load() you will be able to access your data from within your mongo script via the myData object.

data.js

var myData=
[
        {
                "letter" : "A"
        },
        {
                "letter" : "B"
        },
        {
                "letter" : "C"
        }
]

read.js

#!/usr/bin/mongo --quiet

// read data

load('data.js');

// display letters

for (i in myData) {
  var doc  = myData[i];
  print(doc.letter);
}

For writing JSON is easiest to just load your result into a single object. Initialize the object at the beginning with var result={} and then use printjson() at the end to output. Use standard redirection to output the data to a file.

write.js

#!/usr/bin/mongo --quiet

var result=[];

// read data from collection etc...
for (var i=65; i<91; i++) {
  result.push({letter: String.fromCharCode(i)});
}

// output
print("var myData=");
printjson(result);

The shebang lines (#!) will work on a Unix type operating system (Linux or MacOs) they should also work on Windows with Cygwin.

Upvotes: 12

wdberkeley
wdberkeley

Reputation: 11671

No, the mongo shell doesn't have the capability to read and write from files like a fully-fledged programming environment. Use mongoimport, or write the script in a language with an official driver. Node.js will have syntax very close to the mongo shell, although Node.js is an async/event-driven programming environment. Python/PyMongo will be similar and easy to learn if you don't want to deal with structuring the logic to use callbacks.

Upvotes: 1

Related Questions