CODEWITHSUNDEEP
mongodb
More Than Five
More Than Five

Reputation: 10419

Imports to Mongo limited to 16MB if using jsonArray

I am using mongo 2.6.1. I want to import data from a json file > 16 MB. The json is an array of documents. According to their documentation if I use the --jsonArray option, the file can only be 16MB, see http://docs.mongodb.org/manual/reference/program/mongoimport/

Strangely, I have already managed to import data > 16 MB (24MB) no problem using mongoimport, by doing:

mongoimport -db mydb --collection product --file products.json --jsonArray

So what is this 16MB limit then?

Upvotes: 2

Views: 4621

Answers (3)

hitesh chavhan
hitesh chavhan

Reputation: 136

Faced the similar problem I guess the 16MB limitation is still persists with older version. While there is a way around in any case just make your json which has jsonArray into normal json file using linux sed commands which will remove some initial part and end part. You then can import the file with normal mongoimport command.

Upvotes: 0

Leonid Beschastny
Leonid Beschastny

Reputation: 51470

16 MB is a MongoDB BSON document size limit. It means that no document inside MongoDB could exceed 16 MB.

Note that JSON representation of MongoDB document could exceed this limit, since BSON is more compact.

The problem with --jsonArray flag is that mongoimport reads the whole .json file as a single document first, and then performs import on each of its elements, thus suffering from BSON document size limit.

Solution for new MongoDB versions (2.5.x and later)

I just tested mongoimport with latest MongoDB 2.6.4 using very large JSON array (~200 MB) and it worked perfectly.

I'm pretty sure that such an operation was impossible with MongoDB 2.2.x. So, it looks like mongodb.org simply forgot to update mongoimport documentation.

I searched MongoDB bug tracker and found this issue. According to it, this problem was resolved a year ago and the fix was released with MongoDB 2.5.0.

So, feel free to import large JSON documents!

Solution for old MongoDB versions (prior to 2.5.0)

If you're using old version of MongoDB, it's still possible to import large array of documents, using --type json flag instead of --jsonArray. But it assumes a special structure for a file to import from. It's similar to JSON format, except that only one document per line is allowed with no comma after each of them:

{ name: "Widget 1", desc: "This is Widget 1" }
{ name: "Widget 2", desc: "This is Widget 2" }

Upvotes: 10

Ritesh Gupta
Ritesh Gupta

Reputation: 76

Strangely, I have already managed to import data > 16 MB (24MB) no problem using mongoimport, by doing:

If you are happy with the data thats imported this way - you needn't worry about the limit of 16MB. That limit is for each record (document) in the Collection. 16MB in text data is a lot - you could have an entire book in that much space - so its extremely unusual to have a single record more than 16MB in size.

Upvotes: 0

Related Questions