CODEWITHSUNDEEP
pythonmongodbpymongo
Vinodh
Vinodh

Reputation: 916

In PyMongo, The default value of MongoClient's maxPoolSize=100 has no effect?

Below is a simple test snippet for PyMongo connections:

from pymongo import MongoClient

clients = [] 
for i in range(0,1000):
    con = MongoClient('mongodb://localhost:27017/')
    clients.append(con)
    print(i)

Here is the output:

... ... 998 999

And here is the output in MongoDB Console:

2018-03-24T20:00:55.746+0530 I NETWORK [listener] connection accepted from 127.0.0.1:51272 #1500 (1000 connections now open) 2018-03-24T20:00:55.747+0530 I NETWORK [conn1500] received client metadata from 127.0.0.1:51272 conn: { driver: { name: "PyMongo", version: "3.6.1" }, os: { type: "Darwin", name: "Darwin", architecture: "x86_64", version: "10.10.1" }, platform: "CPython 3.6.0.final.0" } 2018-03-24T20:00:55.773+0530 I NETWORK [conn1486] end connection 127.0.0.1:51286 (999 connections now open) 2018-03-24T20:00:55.773+0530 I NETWORK [conn1485] end connection 127.0.0.1:51285 (998 connections now open) 2018-03-24T20:00:55.773+0530 I NETWORK [conn1484] end connection 127.0.0.1:51284 (997 connections now open) . . . . . .

I see MongoClient has a keyword argument maxPoolSize. This has the default value of only 100.

But how the above code works?

Upvotes: 2

Views: 5492

Answers (1)

Wan B.
Wan B.

Reputation: 18835

MongoClient has a keyword argument maxPoolSize. This has the default value of only 100.

That is correct. The default value of maxPoolSize in PyMongo (as of v3.6) MongoClient is 100. However, the maxPoolSize parameter controls the maximum built-in connection pool per MongoDB server for an instance of MongoClient.

Your example snippet creates 1000 MongoClient instances. Please note that you only need one MongoClient for each process, and reuse it for all operations. It is a common mistake to create a new client for each request, which is very inefficient.

But how the above code works?

If you would like to check whether the limit works, you can utilise Python threading module to create threads to test the limit of MongoClient connections.

There are two other parameters that you could utilise:

  • waitQueueMultiple: this parameter sets the limit of how many threads can wait in the queue before getting a connection from the pool. By setting a low maxPoolSize and low waitQueueMultiple your code would receive ExceededMaxWaiters exception.

  • waitQueueTimeoutMS: this parameter sets the time of how long threads should wait in the queue before getting a connection from the pool. By setting a low maxPoolSize and low waitQueueTimeoutMS your code would receive ConnectionFailure exception.

Example: 

def worker(num, db): 
    print("Thread number %s" % num)
    try:
        print(db.test.insert({"a":num, "b":num*2}))
    except Exception as e: 
        print("Exception: %s" % e)
    return 

client = MongoClient("mongodb://localhost:27017/", 
                     maxPoolSize=1, 
                     waitQueueTimeoutMS=1, 
                     waitQueueMultiple=1)
database = client.test
threads = []

for i in range(100): 
    t = threading.Thread(target=worker, args=(i, database))
    threads.append(t) 
    # Start thread.
    t.start()

# Wait for all threads to complete. 
for t in threads: 
    t.join()

See How does connection pooling work in PyMongo? for more details. Please use the default values provided unless you have specific use case that requires alteration of the above values.

Upvotes: 3

Related Questions