CODEWITHSUNDEEP
rubyarraysmongodb
Mr. Demetrius Michael
Mr. Demetrius Michael

Reputation: 2406

MongoDB: How to store large arrays? (ERR Document too large: This BSON documents is limited to 16777216 bytes)

I'm storing large arrays of friends and followers from twitter to MongoDB. I'm surprised I hit the 16MB limit, but I guess it's understandable if I'm grabbing users w/ >2M followers. Is there anyway around this?

I'm doing a social analysis project, and would be looking through these arrays frequently.

Below is my error:

...in `serialize': Document too large: This BSON documents is limited to 16777216 bytes. (BSON::InvalidDocument)

Also using the Mongo Ruby gem, a Ruby answer would be pref :)

Upvotes: 2

Views: 2429

Answers (1)

Gates VP
Gates VP

Reputation: 45277

This will not work as you would like it to. MongoDB does allow for some denormalization, but you cannot denormalize 2M items into a single parent, as you saw with the space concerns.

In SQL, you would implement this as two tables. You can implement this as two collections with MongoDB. It may seem sub-optimal, but it represents the same total amount of work in either MongoDB or SQL.

That stated, MongoDB may not be the "NoSQL" DB you are looking for. Graph Databases like Neo4J are designed for analysis of highly connected data. Also, Redis supports lists without a size limit, so this too may be a better fit.

Upvotes: 1

Related Questions