MongoDB Client throws Wait Queue Full exception

Question

I am seeing an odd issue where The .NET client for MongoDB throws a The wait queue for acquiring a connection to server 127.0.0.1:27017 is full. exception.

I have a semaphore that guards any call to MongoDB, with a size of 10. Meaning, there are never more than 10 concurrent calls to Mongo.

The default connection pool size is 100 for the .NET driver, which is more than 10. so 10 concurrent calls should not be an issue.

To replicate this I have the following code, contrived yes, but it makes the issue visible.

I also found this spec for MongoDB https://github.com/mongodb/specifications/blob/master/source/connection-monitoring-and-pooling/connection-monitoring-and-pooling.rst#id94

Is that related? Does each calling thread (thread pool worker in this case) go into the wait queue and try to grab a connection, and if I have more worker threads, even if concurrency level is low, the connections still have to be assigned to this new calling worker thread?

using System;
using System.Threading.Tasks;
using MongoDB.Bson;
using MongoDB.Driver;
using System.Threading;

namespace ConsoleApp58
{
    public class AsyncSemaphore
    {
        private readonly SemaphoreSlim _semaphore;

        public AsyncSemaphore(int maxConcurrency)
        {
            _semaphore = new SemaphoreSlim(
                maxConcurrency,
                maxConcurrency
            );
        }

        public async Task<T> WaitAsync<T>(Task<T> task)
        {
            await _semaphore.WaitAsync();
            //proves we have the correct max concurrent calls
            //  Console.WriteLine(_semaphore.CurrentCount);

            try
            {
                var result = await task;
                return result;
            }
            finally
            {
                _semaphore.Release();
            }
        }
    }
    
    class Program
    {
        public class SomeEntity
        {
            public ObjectId Id { get; set; }
            public string Name { get; set; }
        }

        static void Main(string[] args)
        {
            var settings = MongoClientSettings.FromUrl(MongoUrl.Create("mongodb://127.0.0.1:27017"));
            // settings.MinConnectionPoolSize = 10;
            // settings.MaxConnectionPoolSize = 1000;
            
            // I get that I can tweak settings, but I want to know why this occurs at all?
            // if we guard the calls with a semaphore, how can this happen?

            var mongoClient = new MongoClient(settings);
            var someCollection = mongoClient.GetDatabase("dummydb").GetCollection<SomeEntity>("some");
            var a = new AsyncSemaphore(10);
            
            // is this somehow related ?
            // https://github.com/mongodb/specifications/blob/master/source/connection-monitoring-and-pooling/connection-monitoring-and-pooling.rst#id94


            _ = Task.Run(() =>
                {
                    while (true)
                    {
                        // this bit is protected by a semaphore of size 10
                        // (we will flood the thread pool with ongoing tasks, yes)
                        _ = a.WaitAsync(RunTask(someCollection))
                            //after the task is done, dump the result
                            // dot is OK, else exception message
                            .ContinueWith(x =>
                            {
                                if (x.IsFaulted)
                                {
                                    Console.WriteLine(x.Exception);
                                }
                            });
                    }
                }
            );

            Console.ReadLine();
        }

        private static async Task<SomeEntity> RunTask(IMongoCollection<SomeEntity> pids)
        {
            //simulate some mongo interaction here
            var res = await pids.Find(x => x.Name == "").FirstOrDefaultAsync();
            return res;
        }
    }
}

Answer 1

Connections take time to be established. You do not instantly get 100 usable connections. If you create a client and immediately request even 10 operations, while there are no available connections, you can hit the wait queue timeout.

Some drivers also had a wait queue length limit. It's not standardized and should be deprecated in my understanding but may continue to exist for compatibility reasons. Consult your driver docs to see how to raise it.

Then, either increase waitQueueTimeoutMS or ramp up the load gradually or wait for connections to be established prior to starting the load (you can use CMAP events for the latter).

Make sure your concurrency bound of 10 outstanding operations is actually working properly too.