CODEWITHSUNDEEP
mongodbmultiprocessingpymongo
Schore
Schore

Reputation: 905

PyMongo and Multiprocessing: ServerSelectionTimeoutError

We recently updated MongoDB from 2.6 to 3.0. Since then, we are having trouble using PyMongo in combination with Multiprocessing.

The issue is that sometimes an operation (e.g. find) within a process hangs for ~30 seconds and then throws an exception "ServerSelectionTimeoutError: No servers found yet".

The behavior seems independent from the input, as our script usually runs just fine for a few times and then hangs randomly.

The log files do not show any entries related to timeouts, nor did I find any useful information on the Internet about this issue.

The script is running in our test environment, meaning there are no replica sets involved and the Mongo instance in bound to localhost.

Here is the stack trace for completeness: 

   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "somescript.py", line 109, in run
    self.find_incoming_cc()
  File "somescript.py", line 370, in find_incoming_cc
    {'_id': 1, 'cc': 1}
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 983, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 908, in _refresh
    self.__read_preference))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 813, in __send_message
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 728, in _send_message_with_response
    server = topology.select_server(selector)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/topology.py", line 121, in select_server
    address))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/topology.py", line 97, in select_servers
    self._error_message(selector))
ServerSelectionTimeoutError: No servers found yet

Now for the question: Is there any known issue/bug when using PyMongo with Multiprocessing? Is there a way to debug the exception?

Thanks for any help!

Upvotes: 1

Views: 2274

Answers (1)

VigneshChennai
VigneshChennai

Reputation: 576

It is bug in pymongo version 3.0.x. Bug report url https://jira.mongodb.org/browse/PYTHON-961

Workaround for this issue. (Tested in pymongo 3.0.3) Pass “connect=False” in MongoClient object initialisation

MongoClient(uri, connect=False)

Or simply wait for few seconds before creating instance of MongoClient in the child process (like time.sleep(2)).

def start(uri):
  time.sleep(2)
  mclient = MongoClient(uri)
  mclient.db.collection.find_one()

if __name__ == '__main__':
  p = multiprocessing.Process(target=start, args=('mongodb://localhost:27017/',))
  p.start()

Upvotes: 2

Related Questions