Elasticsearch solo vs Couchbase+Elasticsearch

Question

Let me describe the issue. Now I'm at the stage of architectural changes. I used to use Elasticsearch as aggregation and search tool, but now I'm thinking about use it as main DB. I read that there is not recommended to use Elasticsearch as a main DB, only as a index. So I read presentation about connection it with Couchbase:

https://2013.nosql-matters.org/bcn/wp-content/uploads/2013/12/nosql13-bcn-couchbase-elasticsearch-jeroen-reijn.pdf

I downloaded and installed Couchbase and plugin for integration with Elasticsearch

https://github.com/couchbaselabs/elasticsearch-transport-couchbase

I'm not convinced about this solution. Firstly, aggregation is faster in Elasticsearch. Secondly, Elasticsearch supports histogram, geopoints, full text search and many other, so it seems that only reason to keep Couchbase is not to have Elasticsearch as the main DB.

So my question is about the benefits of integration Elasticsearch with Couchbase in my case?

Answer 1

The only significant issue I've run into so far with using elasticsearch as a primary data store is the indexing delay when you index a new document. Elasticsearch is a "near" realtime search engine...the "near" qualifier is necessary because there is up to a 1-sec delay after you index a document before it will be found during a search.

Here's a link to the relevant portion of the elasticsearch guide:

Near real-time search in elasticsearch

This was an issue in a web application I built that contains a page that lists users. My scenario was the admin clicked a "new user" button on a user list page and he was then brought to another page to create the user. When the admin saved the user document, he was redirected to the list page, but the newly created user did not show up because of elasticsearch's indexing delay.

The elasticsearch guide says you can manually refresh the index, but says not to do it in production.

...don’t do a manual refresh every time you index a document in production; it will hurt your performance. Instead, your application needs to be aware of the near real-time nature of Elasticsearch and make allowances for it.

I ended up refreshing the index anyway because creating new users is a very rare occurrence in my app, but it's not a very good solution.

I posted a question a couple of months ago asking how other people get around this issue:

How to deal with Elasticsearch index delay

The answer to that question had a suggestion I liked. Basically the author suggests inserting the record into the list manually using the data you submit instead of waiting for the return from the server. That should work as long as you don't rely on server generated fields.

Ultimately though, you should not encounter this issue with a database like Couchbase.

Answer 2

Using the couchbase plugin will allow changes to data in Couchbase to automatically be (re)indexed into Elasticsearch. However, you may want to look at whether the schema/mapping that Couchbase may impose is appropriate to your queries. Once the data is in Elasticsearch, you can still query it using the Elasticsearch queries directly to do things like searches or aggregations.

One benefit to using Couchbase may be that it allows for cross data-center replication (using multiple elasticsearch clusters) whereas normally having an Elasticsearch across multiple data centers is not recommended practice.

Answer 3

I am using ES as a data base for a few years now, never had a problem whereas I tried also MongoDB with lot of issues!

It's always good to have a backup of data, in your case with "Couchbase", but not necessary. You can create ES snapshot every day, that could be enough.

A point to consider is ES is "near real time", means data is available after a few times (milli-seconds) after edition.

ES is fast, easy to setup & scale very well. HTTP protocol is the best way to communicate in any languages.