CODEWITHSUNDEEP
pythondjangocelerymongoenginedjango-celery
Ouss
Ouss

Reputation: 3865

MongoEngine and dealing with "UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking"

I am using Celery and MongoEngine as part of my Django App with.

I am getting this warning, when a celery @shared_task accesses the mongodb database via mongoengine model classes:

UserWarning: MongoClient opened before fork. Create MongoClient with
connect=False,or create client after forking. See PyMongo's
documentation for details:
http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing

It clearly has something to do with multiprocessing and pyMongo that is that mongoengine is based on.

My question is:

What is the best strategy to avoid this issue with mongoengine?

Please note that i am connecting to the mongodb with mongoengine in the settings.py:

mongoengine.connect('my_mongo_database_name', alias='default')

Upvotes: 8

Views: 4552

Answers (2)

Jose Enrique
Jose Enrique

Reputation: 461

Ok this is what i learned:

  1. Celery has "subworkers", when you run celery -A app worker it launches 1 worker with many subworkers, the number of subworkers by default is equal to the number of cores in your processor
  2. Celery by default uses prefork mechanism to spawn "subworkers", which means it uses fork to create a new process for each "subworker"
  3. I was initializing my mongo db connection on the signal @signals.worker_init.connect, but it seems that such signal is executed on the parent process, before doing the forking.
  4. To initialize the mongo db connection AFTER the forking, i needed to use the signal @signals.worker_process_init.connect instead.

It looks like this:

@signals.worker_process_init.connect
def on_worker_process_init(**kwargs):
    DatabaseConnection()

Upvotes: 1

Ouss
Ouss

Reputation: 3865

After searching a little bit online, i found out that it is possible to pass additional arguments to the mongoengine.connect function the additional arguments will be passed to the underlying PyMongo classes & functions.

So i simply edited the mongoengine.connect() call to the following:

mongoengine.connect('my_mongo_database_name', alias='default', connect=False)

And the warning stopped appearing. None the less i am not sure this is the best way to deal with the warning. If you have a better answer please post it and i will gladly test it and eventually accept it.

Upvotes: 10

Related Questions