CODEWITHSUNDEEP
mongodb
GrinnS
GrinnS

Reputation: 143

MongoDB import piece of json files in collection

I have a folder containing .json files. Each file looks like this :

{"data":
   [{"status":0,
     "timestamp":...
     },
     {"status":0,
     "timestamp":...
     },{...
    }],
  "otherinfos":"random"
}

What I need to do is to import in a collection the content of the data array. File are about 10Mo, and folder contains thousands of files... I tried "mongoimport" command, but it inserted the whole object instead of the desired list of element available in the "data" array. Is there a way to import an array from a json file in mongo ? Is there an out-of-the-box solution to import multiple files from folder ?

Thank you,

Antoine

PS : I did a small java software to read each file, and import the list of object from the array in mongo but after weeks of computation I would love to hear better solutions.

Edit : I would like my mongoDB data to be stored like this, with each status and timestamp as a separate document : 

{
  "status":0,
  "timestamp":...
},
{
  "status":0,
  "timestamp":...
},
{...
}

So I can request on a timestamp and get the status for example. I have a billion entries like that.

Upvotes: 1

Views: 942

Answers (1)

Vince Bowdren
Vince Bowdren

Reputation: 9208

You could do it as a two-stage process:

  1. Use mongoimport to load the data from each json file into a temporary holding collection; this should be fairly quick.
  2. Use aggregation to transform the data into discrete documents, saving each one to a new collection with the $out operator

The aggregation command might look something like this:

db.tempcollection.aggregate([
  { $unwind: "$data" },
  { $project: {
    _id: 0, 
    "status" : "$data.status", 
    "timestamp" : "$data.timestamp"
    }
  },
  { $out: "newcollection" }
]);

This is likely to be processed fairly quickly, because it can be done by MongoDB itself on the database server; however it may be worth doing some performance testing on a small data sample to find out.

Upvotes: 1

Related Questions