CODEWITHSUNDEEP
mongodbpython-3.xpymongopython-multithreading
pslover
pslover

Reputation: 72

Error while retrieving result in pymongo

I have an python application which creates number of threads for a job. each thread connects to mongodb and retrieve data. Number of allowed connection to mongodb is 200 which I'm taking care using semaphore. And once mongo querying job is done each thread closes mongodb connection. But while executing this application I'm getting same error for all threads. Error is:

Traceback (most recent call last):
  File "C:\Python34\lib\threading.py", line 921, in _bootstrap_inner
    self.run()
  File "C:\Python34\lib\threading.py", line 869, in run
    self._target(*self._args, **self._kwargs)
  File "C:/path/pytest/under_construction/testAlgo.py", line 95, in sample_thread
    status=monObj.process_status(list_value1,list_value2,5,120,120)
  File "C:\path\pytest\under_construction\mongo_lib.py", line 153, in process_status
    result=self.mongo_result('Submission','find',q={})
  File "C:\path\pytest\under_construction\mongo_lib.py", line 53, in mongo_result
    result=list(_query[query_type.lower()](query_string[keys]))
  File "C:\Python34\lib\site-packages\pymongo\cursor.py", line 1076, in __next__
    if len(self.__data) or self._refresh():
  File "C:\Python34\lib\site-packages\pymongo\cursor.py", line 1037, in _refresh
    limit, self.__id))
  File "C:\Python34\lib\site-packages\pymongo\cursor.py", line 933, in __send_message
    res = client._send_message_with_response(message, **kwargs)
  File "C:\Python34\lib\site-packages\pymongo\mongo_client.py", line 1205, in _send_message_with_response
    response = self.__send_and_receive(message, sock_info)
  File "C:\Python34\lib\site-packages\pymongo\mongo_client.py", line 1182, in __send_and_receive
    return self.__receive_message_on_socket(1, request_id, sock_info)
  File "C:\Python34\lib\site-packages\pymongo\mongo_client.py", line 1174, in __receive_message_on_socket
    return self.__receive_data_on_socket(length - 16, sock_info)
  File "C:\Python34\lib\site-packages\pymongo\mongo_client.py", line 1153, in __receive_data_on_socket
    chunk = sock_info.sock.recv(length)
MemoryError

Code for creating mongo connection

client=MongoClient(mc_name,port)

I was thinking, is this error due to results of all threads accumulating at one port of machine running my application?

Upvotes: 1

Views: 677

Answers (1)

JohnnyHK
JohnnyHK

Reputation: 312179

MongoClient is a thread-safe connection pool, so you should be creating a single instance that's shared by all the worker threads rather than having each thread create its own.

The connection pool size defaults to 100, but if you want to make it even larger you can use the maxPoolSize parameter to do that (e.g. maxPoolSize=200).

Upvotes: 1

Related Questions