CODEWITHSUNDEEP
mongodbpymongomongoengine
CandyCrusher
CandyCrusher

Reputation: 316

How to handle PyMongo/MongoEngine AutoReconnect?

When I used MongoEngine to query over 100K documents iteratively in a collection. It usually ended up with "pymongo.errors.AutoReconnect: [Errno 54] Connection reset by peer".

I've already checked other answer/solution for AutoReconnect but I doubt if those are the same case as the one I met. So I post this question.

Is there any tips to avoid AutoReconnect or anything I can do for diagnosis?

Here are some solutions I have collected:

  1. use try...catch...
  2. use c extension (pymongo_has_c())
  3. use third-party library like MongoDBProxy
  4. set keyword argument like timeout and bulk_size(I tried but it didn't work)

Upvotes: 2

Views: 1722

Answers (1)

Datalker
Datalker

Reputation: 366

MongoProxy didn't work for me out of the box with Mongo engine. And using try .. except everywhere is annoying. Finally I ended up with such quick solution. Hope it helps.

import logging

from mongoengine import *

class SafeDocumentMixin:

    def save_safe(self, *args, **kwargs):
        for attempt in range(5):
            try:
                return self.save(*args, **kwargs)
            except pymongo.errors.AutoReconnect as e:
                wait_t = 0.5 * pow(2, attempt) # exponential back off
                l.warning("PyMongo auto-reconnecting... %s. Waiting %.1f seconds.", str(e), wait_t)
                time.sleep(wait_t)

    @classmethod
    def objects_safe(cls, *args, **kwargs):
        for attempt in range(5):
            try:
                return cls.objects(*args, **kwargs)
            except pymongo.errors.AutoReconnect as e:
                wait_t = 0.5 * pow(2, attempt) # exponential back off
                logging.warning("PyMongo auto-reconnecting... %s. Waiting %.1f seconds.", str(e), wait_t)
                time.sleep(wait_t)

class Person(Document, SafeDocumentMixin):
    name = StringField()
    age = IntField()

And using like this

Person.objects_safe(age='23')

Probably even more solution would be to do monkey patching but I prefer to see that I'm calling a custom method

Upvotes: 2

Related Questions