CODEWITHSUNDEEP
mongodbdocumentphp-mongodb
Salvador Dali
Salvador Dali

Reputation: 222471

mongodb limit in the embedded document

I need to create a message system, where a person can have a conversation with many users. For example I start to speak with user2, user3 and user4, so anyone of them can see the whole conversation, and if the conversation is not private at any point of time any of participants can add any other person to the conversation.

Here is my idea how to do this. I am using Mongo and my idea is to use dialog as an instance instead of message.

The schema is listed as follows:

{
_id : ...., // dialog Id
'private' : 0 // is the conversation private
'participants' : [1, 3, 5, 6], //people who are in the conversation
'msgs' :[
  {
   'mid' : ...// id of a message
   'pid': 1, // person who wrote a message
   'msg' : 'tafasd' //message
  },
  ....
  {
   'mid' : ...// id of a message
   'pid': 1, // person who wrote a message
   'msg' : 'tafasd' //message
  }
]
}

I can see some pros for this approach - in a big database it will be easy to find messages for some particular conversation. - it will be easy to add people to the conversation.

but here is a problem, for which I can't find a solution: the conversation is becoming too long (take skype as an example) and they are not showing you all the conversation, they are showing you a part and afterwards they are showing you additional messages. In other situations skip, limit solves the case, but how can I do this here?

If this is impossible what suggestions do you have?

Upvotes: 7

Views: 7425

Answers (3)

Ashique Desai
Ashique Desai

Reputation: 370

I think one can use the subset pattern here to avoid running out of the 16GB limit of a document as alluded by @lig here. Here's how you would model it with the subset pattern:

dialogs collection:

{
 _id : ...., // dialog Id
 'private' : 0 // is the conversation private
 'participants' : [1, 3, 5, 6], //people who are in the conversation
 'messages' :[
  // Most recent messages embedded here (example: latest 100 messages)
  {
   'mid' : ...// id of a message
   'pid': 1, // person who wrote a message
   'msg' : 'How are you doing?' //message
  },
  ....
  {
   'mid' : ...// id of a message
   'pid': 2, // person who wrote a message
   'msg' : 'I am fine buddy!' //message
  }
 ]
}

Notice, this is identical to the original example, except we do not embed all the messages in the dialogue document as before, we only embed the subset of them (most recent) and store all the older ones including the most recent in its own collection - messages collection - like below:

messages collection:

{
  'mid' : ...// id of a message
  'pid': 1, // person who wrote a message
  'msg' : 'How are you doing?' //message
}

By keeping only the most frequently accessed messages directly within the dialogs documents, we reduce the working set and improve the performance. Meanwhile, the older messages peacefully reside in their own collection, ready to be fetched whenever needed.

One tradeoff that we must make when using the subset pattern is that we must manage the subset and also if we need to pull in older reviews or all of the information, it will require additional trips to the database to do so.

Additional resources:

I have written an article on using the Subset pattern on Medium here: https://medium.com/@desai.ashique/data-modelling-for-many-to-many-relationship-in-mongodb-48f1c80910b7

MongoDB blog on Subset Pattern: https://www.mongodb.com/blog/post/building-with-patterns-the-subset-pattern

Upvotes: 0

jmacinnes
jmacinnes

Reputation: 1609

The MongoDB docs explain how to select a subrange of an array element.

db.dialogs.find({"_id": [dialogId]}, {msgs:{$slice: 5}}) // first 5 comments
db.dialogs.find({"_id": [dialogId]}, {msgs:{$slice: -5}}) // last 5 comments
db.dialogs.find({"_id": [dialogId]}, {msgs:{$slice: [20, 10]}}) // skip 20, limit 10
db.dialogs.find({"_id": [dialogId]}, {msgs:{$slice: [-20, 10]}}) // 20 from end, limit 10

You can use this technique to only select the messages that are relevant to your UI. However, I'm not sure that this is a good schema design. You may want to consider separating out "visible" messages from "archived" messages. It might make the querying a bit easier/faster.

Upvotes: 17

lig
lig

Reputation: 3890

There are caveats if your conversation will have many many messages:

  1. You will notice significant performance reduction on slicing messages arrays as mongodb will do load all of them and will slice the list before return to driver only.
  2. There is document size limit (16MB for now) that could be possibly reached by this approach.

My suggestions is:

  1. Use two collections: one for conversations and the other for messages.
  2. Use dbref in messages to conversation (index this field with the message timestamp to be able to select older ranges on user request).
  3. Additional use separate capped collection for every conversation. It will be easy to find it by name if you build it like "conversation_"

Result:

  • You will have to write all messages twice. But into separate collections which is normal.
  • When you want to show your conversation you will need just to select all the data from one collection in natural sort order which is very fast.
  • Your capped collections will automatically store last messages and delete old.
  • You may show older messages on the user request by querying main messages collection.

Upvotes: 3

Related Questions