CODEWITHSUNDEEP
pythonmongodbmongodb-querymongoengine
Rob
Rob

Reputation: 3459

how to speed up mongoengine queries

If I have two objects like:

class User(Document):
    name = StringField()
    following = ListField(ReferenceField('User'))
    meta = {
        'indexes': [
            'following',
        ]
    }

class Media(Document):
    owner = ReferenceField('User')
    url = StringField()
    is_hidden = BooleanField()
    posted_date = Date

    meta = {
        'indexes': [
            'owner',
            'posted_date',
            'is_hidden',
        ]
    }

and when I want to look up the following condition where it is not hidden and its owner is someone I am following and it was recently posted, I have a query like this:

user = User.objects.first()
Media.objects(Q(owner__in=user.following) &
              Q(is_hidden=False) &
              Q(posted_date__gte=dt.now()-dt.timedelta(days=3))

which is not scaling and is becoming much slower. What can I do to speed up the performance of these types of complex queries?

Upvotes: 1

Views: 2571

Answers (2)

Kristian
Kristian

Reputation: 756

If you plan to use mongoengine in production with large documents take a look at this post: https://github.com/MongoEngine/mongoengine/issues/1230

We were using mongoengine but it became really slow for the reason explained in the post above.

We ended up rewriting our whole backend using https://github.com/mongodb/pymodm That is a relatively new project for the mongodb team. We are using it with Django and it is working much faster than mongoengine with exactly the same database.

Upvotes: 1

sergiuz
sergiuz

Reputation: 5529

1) Use User.objects.get(id=user_id) instead of first(). Tough I am not really sure if that will make a difference, I think yes and that is a find operation where MongoDB will return a cursor and mongoengine iterates to the first document. Instead, get() is doing a findOne and returns only 1 document. If I mistake, someone should correct me please.

2) Use a compound index (not multiple single index) since your query is using multiple fields (example here also):

 meta = {
            'indexes': [
                ('owner', 'posted_date', 'is_hidden',)
            ]
        }

3) Limit returning data to only the fields you need, project your fields, using only()

Use explain() on your query to styudy it and improve it in oder to reach a covered query.

Upvotes: 1

Related Questions