CODEWITHSUNDEEP
jsonmongodbmongoimportdatabase
amber4478
amber4478

Reputation: 6743

Importing large json files into MongoDB

So I have a json file that has multiple records, looking like this...

{"2597401":[{
"jobID":"2597401",
"account":"TG-CCR120014",
"user":"c",
"pkgT":{"pgi/7.2-5":{"libA":["libpgc.so"],"flavor":["default"]}},
"startEpoch":"1338497979",
"runTime":"1022",
"execType":"user:binary",
"exec":"/share/home/01482/c/appker/ranger/NPB3.3.1/NPB3.3-MPI/bin/ft.D.64",
"numNodes":"4",
"sha1":"5a79879235aa31b6a46e73b43879428e2a175db5",
"execEpoch":1336766742,
"execModify":"Fri May 11 15:05:42 2012",
"startTime":"Thu May 31 15:59:39 2012",
"numCores":"64",
"sizeT":{"bss":"1881400168","text":"239574","data":"22504"}},

{"jobID":"2597401",
"account":"TG-CCR120014",
"user":"c",
"pkgT":{"pgi/7.2-5":{"libA":["libpgc.so"],"flavor":["default"]}},
"startEpoch":"1338497946",
"runTime":"33",
"execType":"user:binary",
"exec":"/share/home/01482/c/appker/ranger/NPB3.3.1/NPB3.3-MPI/bin/cg.C.64",
"numNodes":"4",
"sha1":"caf415e011e28b7e4e5b050fb61cbf71a62a9789",
"execEpoch":1336766735,
"execModify":"Fri May 11 15:05:35 2012",
"startTime":"Thu May 31 15:59:06 2012",
"numCores":"64",
"sizeT":{"bss":"29630984","text":"225749","data":"20360"}},

{"jobID":"2597401",
"account":"TG-CCR120014",
"user":"c",
"pkgT":{"pgi/7.2-5":{"libA":["libpgc.so"],"flavor":["default"]}},
"startEpoch":"1338500447",
"runTime":"145",
"execType":"user:binary",
"exec":"/share/home/01482/appker/ranger/NPB3.3.1/NPB3.3-MPI/bin/mg.D.64",
"numNodes":"4",
"sha1":"173de32e1514ad097b1c051ec49c4eb240f2001f",
"execEpoch":1336766756,
"execModify":"Fri May 11 15:05:56 2012",
"startTime":"Thu May 31 16:40:47 2012",
"numCores":"64",
"sizeT":{"bss":"456954120","text":"426186","data":"22184"}},

{"2597401":[{....

Each record is on a single line. So the general format of the json file is as follows:

{"1111111: [{"jobID":value,"account":value,"user":value,"pkgT":{value:{"libA":[VALUES],"flavor":[value]}},"startEpoch":value,"runTime":value,"execType":value,"exec":value,"numNodes":value,"sha1":value,"execEpoch":value,"execModify":value,"startTime":value,"numCores":value,"sizeT":{"bss":value,"text":value,"data":value}}

I am trying to use mongoimport to import these into a database using the following command:

mongoimport --db dbName --collection collectionName fileName --jsonArray

I read that each record should be on it's own line, which it is. However, when I import the file it imports it as a single document (aka record). How can I get mongo to interpret this file as containing multiple records instead of a single huge document? Thanks for your help!!!

Upvotes: 1

Views: 3039

Answers (1)

amber4478
amber4478

Reputation: 6743

I was able to fix it. I had curly brackets surrounding all the text in the file. When I removed them, it successfully imported the records individually.

Upvotes: 1

Related Questions