What is the best method to seeding a Node / MongoDB application?

Question

So, I'm new to the MEAN stack, and I've hit a wall trying to seed MongoDB. I'm using Mongoose to communicate with the database, and there's a bunch of documentation suggesting I should be able to seed using populated JSON files.

What I've tried:

node-mongo-seed; Pretty straight forward, but consistently throws errors on the end of arrays. (Perhaps the missing bson module is at fault?)

{ [Error: Cannot find module '../build/Release/bson'] code: 'MODULE_NOT_FOUND' }
js-bson: Failed to load c++ bson extension, using pure JS version
Seeding files from directory /Users/Antwisted/code/wdi/MEAN/seeds
----------------------
Seeding collection locations
err =  [SyntaxError: /Users/Antwisted/code/wdi/MEAN/seeds/locations.json: Unexpected token {]

mongoose-seed; Also pretty straight forward, basically puts the JSON objects into a variable before exporting to the database. Promising, but... more errors...

Successfully initialized mongoose-seed
[ 'app/models/locationsModel.js' ]
Locations collection cleared
Error creating document [0] of Location model
Error: Location validation failed
Error creating document [1] of Location model
Error: Location validation failed
Error creating document [2] of Location model
Error: Location validation failed...

So, my thoughts were that it was probably a syntax error within the JSON structure, but playing around with that has not yielded any real solutions (or maybe I'm missing it?). Sample of my JSON:

{
    {
        "header": "Dan's Place",
        "rating": 3,
        "address": "125 High Street, New York, 10001",
        "cord1": -73.0812,
        "cord2": 40.8732,
        "attributes": ["Hot drinks", "Food", "Premium wifi"],
        "hours": [
            {
                "days": "Monday - Friday",
                "hours": "7:00am - 7:00pm",
                "closed": false
            },
            {
                "days": "Saturday",
                "hours": "8:00am - 5:00pm",
                "closed": false
            },
            {
                "days": "Sunday",
                "closed": true
            }
        ],
        "reviews": [
            {
                "rating": 4,
                "id": ObjectId(),
                "author": "Philly B.",
                "timestamp": "new Date('Feb 3, 2016')",
                "body": "It was fine, but coffee was a bit dull. Nice atmosphere."
            },
            {
                "rating": 3,
                "id": ObjectId(),
                "author": "Tom B.",
                "timestamp": "new Date('Feb 23, 2016')",
                "body": "I asked for her number. She said no."
            }
        ]
    },
    {
        "header": "Jared's Jive",
        "rating": 5,
        "address": "747 Fly Court, New York, 10001",
        "cord1": -73.0812,
        "cord2": 40.8732,
        "attributes": ["Live Music", "Rooftop Bar", "2 Floors"],
        "hours": [
            {
                "days": "Monday - Friday",
                "hours": "7:00am - 7:00pm",
                "closed": false
            },
            {
                "days": "Saturday",
                "hours": "8:00am - 5:00pm",
                "closed": false
            },
            {
                "days": "Sunday",
                "closed": true
            }
        ],
        "reviews": [
            {
                "rating": 5,
                "id": ObjectId(),
                "author": "Jacob G.",
                "timestamp": "new Date('Feb 3, 2016')",
                "body": "Whoa! The music here is wicked good. Definitely going again."
            },
            {
                "rating": 4,
                "id": ObjectId(),
                "author": "Tom B.",
                "timestamp": "new Date('Feb 23, 2016')",
                "body": "I asked to play her a tune. She said no."
            }
        ]
    }
}

Additionally, I'm not entirely sure how to specify subdocuments within the JSON (assuming I can get the seeding process to work correctly in the first place).

Here's my model:

var mongoose = require('mongoose');

var subHoursSchema = new mongoose.Schema({
    days: {type: String, required: true},
    opening: String,
    closing: String,
    closed: {type: Boolean, required: true}
});

var subReviewsSchema = new mongoose.Schema({
    rating: {type: Number, required: true, min: 0, max: 5},
    author: String,
    timestamp: {type: Date, "default": Date.now},
    body: String
}); 

var locationSchema = new mongoose.Schema({
    name: {type: String, required: true},
    address: String,
    rating: {type: Number, "default": 0, min: 0, max: 5}, 
    attributes: [String],
    coordinates: {type: [Number], index: '2dsphere'},
    openHours: [subHoursSchema],
    reviews: [subReviewsSchema]
});

mongoose.model('Location', locationSchema);

Any insight on how to navigate these issues would be greatly appreciated. Thanks!

Answer 1

I would also recommend looking into mongo-seeding. There is both a JS library version and a CLI version. The motivation for this library is described here.

Answer 2

I solved this issue on a project by dumping the relevant data to an extended JSON array formatted file using mongoexport --jsonArray, then importing this back into POJO format inside the Node application using the EJSON package. I then just use Mongoose to insert the resulting JS array back into the database using the correct collection model you've created using Mongoose.

The necessary JSON data files to seed the application for a first-run are checked into the application repository. Here's a quick sample you may be able to adapt to your purposes:

// ...
// 'Items' is the Mongoose collection model.
const itemResult = await Items.find({}).exec();
if(itemResult.length === 0) {
  const itemsSeedDataRaw = fs.readFileSync(`${__dirname}/data/items.json`, 'utf8');
  const itemsSeedData = EJSON.parse(itemsSeedDataRaw);
  await Items.insertMany(itemsSeedData);
}
// ...

Answer 3

You JSON is not flowing your schema.

Fix your JSON to this:

{
    {
        "name": "Dan's Place",
        "rating": 3,
        "address": "125 High Street, New York, 10001",
        "coordinates": [-73.0812, 40.8732],
        "attributes": ["Hot drinks", "Food", "Premium wifi"],
        "openHours": [
            {
                "days": "Monday - Friday",
                "opening": "7:00am",
                "closing": "7:00pm",
                "closed": false
            },
            {
                "days": "Saturday",
                "opening": "8:00am",
                "closing": "5:00pm",
                "closed": false
            },
            {
                "days": "Sunday",
                "closed": true
            }
        ],
        "reviews": [
            {
                "rating": 4,
                "author": "Philly B.",
                "timestamp": "new Date('Feb 3, 2016')",
                "body": "It was fine, but coffee was a bit dull. Nice atmosphere."
            },
            {
                "rating": 3,
                "author": "Tom B.",
                "timestamp": "new Date('Feb 23, 2016')",
                "body": "I asked for her number. She said no."
            }
        ]
    },
    {
        "name": "Jared's Jive",
        "rating": 5,
        "address": "747 Fly Court, New York, 10001",
        "coordinates": [-73.0812, 40.8732],
        "attributes": ["Live Music", "Rooftop Bar", "2 Floors"],
        "openHours": [
            {
                "days": "Monday - Friday",
                "opening": "7:00am",
                "closing": "7:00pm",
                "closed": false
            },
            {
                "days": "Saturday",
                "opening": "8:00am",
                "closing": "5:00pm",
                "closed": false
            },
            {
                "days": "Sunday",
                "closed": true
            }
        ],
        "reviews": [
            {
                "rating": 5,
                "author": "Jacob G.",
                "timestamp": "new Date('Feb 3, 2016')",
                "body": "Whoa! The music here is wicked good. Definitely going again."
            },
            {
                "rating": 4,
                "author": "Tom B.",
                "timestamp": "new Date('Feb 23, 2016')",
                "body": "I asked to play her a tune. She said no."
            }
        ]
    }
}

You can use mongoose-data-seed to write your own seed script that interacting your mongoose models with: https://github.com/sharvit/mongoose-data-seed

Answer 4

You can populate MongoDB in the CLI using mongoimport

It will load a JSON file into a specified MongoDB Instance & Collection, all you need is for a mongod instance to be running before executing.

Here is a walkthrough of using mongoimport.