CODEWITHSUNDEEP
c#.netazureamazon-ec2couchbase
Mark Vincze
Mark Vincze

Reputation: 8043

Why can opening a Couchbase bucket be very slow?

I'm trying to use Couchbase from .NET with the official SDK. I'm hosting the Couchbase cluster in Amazon EC2.
The machine I'm trying to connect from is hosted in Microsoft Azure.

For some reason, opening a bucket is extremely slow, this afternoon it took ~3 seconds to open a bucket, but now for some reason it's more than 10 seconds.
The memory and CPU utilization of the Couchbase servers are very low.

My configuration:

<couchbaseClients>
  <couchbase useSsl="false" operationLifespan="1000">
    <servers>
      <!-- Ip addresses obscured... -->
      <add uri="http://1.1.1.1:8091/pools"></add>
      <add uri="http://1.1.1.2:8091/pools"></add>
    </servers>
  </couchbase>
</couchbaseClients>

The code I'm trying:

var cluster = new Cluster("couchbaseClients/couchbase");

using (var bucket = cluster.OpenBucket("bucketname")) // This is taking 10-50 seconds.
{
    var obj = new TestClass { };

    // This is fast
    var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}

using (var bucket = cluster.OpenBucket("bucketname")) // This is taking 10-50 seconds.
{
    var obj = new TestClass { };

    // This is fast
    var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}

using (var bucket = cluster.OpenBucket("bucketname")) // This is taking 10-50 seconds.
{
    var obj = new TestClass { };

    // This is fast
    var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}

I tried to optimize by using ClusterHelper.GetBucket() instead of cluster.OpenBucket(), but the first GetBucket() is still very slow (everything else is fast in that case).

I experimented with other calls on the bucket (Get, Contains, etc.), and everything is fast, only opening the bucket itself is slow.

How can I troubleshoot what the problem is?

UPDATE: I set up logging based on @jeffrymorris' suggestion, and the I see the following error messages at the point where we spend a lot of time:

DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Bootstrapping with 127.0.0.1:11210
INFO Couchbase.IO.ConnectionPool... Node 127.0.0.1:11210 failed to initialize, reason: System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time...

And after this there are three other exceptions about trying to connect to 127.0.0.1:11210, and there are seconds passed between those messages, so this is causing the slowness.
Why is the client trying to connect to localhost? I don't have that anywhere in my configuration.

I debugged the code of couchbase-net-client, and this is happening when I call OpenBucket() in CarrierPublicationProvider.GetConfig(). It gets the bucket configuration object, and for some reason it contains not one of the servers I configured in the web.config, but localhost:8091 instead. I'm still trying to figure out why this is happening.

(This is probably not happening on my dev machine, because there I have a local Couchbase server installation. If I stop that, then it gets slow there as well, and I'm seeing the same Bootstrapping with 127.0.0.1:11210 error messages in the logs.)

Upvotes: 2

Views: 2238

Answers (1)

Mark Vincze
Mark Vincze

Reputation: 8043

Figuring out the problem took some debugging of the Couchbase client code, and looking at the logs (thanks for the tip, @jeffrymorris!).

It turns out that if you don't have any preconfigured bucket, then the client will initially try to connect to localhost:8091 (event if that's not in your servers section), which can take seconds until it realizes there is no server there. This doesn't happen if you have any bucket configured.

So I had to change my configuration from this:

<couchbaseClients>
  <couchbase useSsl="false" operationLifespan="1000">
    <servers>
      <!-- Ip addresses obscured... -->
      <add uri="http://1.1.1.1:8091/pools"></add>
      <add uri="http://1.1.1.2:8091/pools"></add>
    </servers>
  </couchbase>
</couchbaseClients>

To this:

<couchbaseClients>
  <couchbase useSsl="false" operationLifespan="1000">
    <servers>
      <!-- Ip addresses obscured... -->
      <add uri="http://1.1.1.1:8091/pools"></add>
      <add uri="http://1.1.1.2:8091/pools"></add>
    </servers>
    <buckets>
      <add name="default" useSsl="false" operationLifespan="1000">
      </add>
    </buckets>
  </couchbase>
</couchbaseClients>

Note that I'm not actually going to use a bucket called default, but having the configuration there sill prevent the client from trying to connect to localhost. (This is happening in the method ConfigProviderBase.GetOrCreateConfiguration.)

Update: The issue has been fixed here: https://github.com/couchbase/couchbase-net-client/commit/020093b422a78728dd49d75d9fe9f1e00d01a0f2

Upvotes: 2

Related Questions