CODEWITHSUNDEEP
elasticsearchanalyticsbusiness-intelligencedbt
denim
denim

Reputation: 463

Elastic Search - Is it the appropriate DB for an Analytics team?

I am a member of an Analytics team that recently moved it's Data Warehouse into Elastic Search. The DW is accessed through Dremio.

However, I am having second thoughts regarding whether Elastic Search is the appropriate DB for an Analytics team that performs a lot of day-to-day Analytics. I would prefer we kept our DW in one of BigQuery/Snowflake/Redshift and use "dbt" tool for transforming data and writing it back into the DB.

I can't find a "dbt"-like tool to perform quick data transformations after reading from Elastic Search and Dremio is not mature enough tool for that. I would like to solicit your thoughts on Elastic Search and whether is an appropriate DB for day-to-day analytics.

I appreciate your responses.

Edit: I work at an online retailer. Our data is not "big data" in any sense. In the order of a few thousand orders per day. Most of our work is responding to inquiries from various teams/departments. Some of these questions go beyond a simple query. We have to build customized data marts that involve multiple steps in between. As a result, we need a tool that would allow us to transform data quickly and put the result set into a database. One such tool is "dbt" but it doesn't support Elastic Search. So the question is whether there is an appropriate tool for this job or Elastic Search is not appropriate for our use case.

Upvotes: 4

Views: 1852

Answers (2)

Jlam
Jlam

Reputation: 1920

Depends on what specific metrics you need, ES aggregation can support a lot of basic metrics. For cost considerations and less infra to support and reducing complexity, I generally advise companies to start with that before over engineering or prematurely optimizing

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html

Upvotes: 0

Vitaliy Fedorchenko
Vitaliy Fedorchenko

Reputation: 9245

Taking into account

Our data is not "big data" in any sense.

most likely ElasticSearch is not appropriate choice. Only reason to use ES is a lot of search-like queries with 'contains' filtering over text-type fields and only if dataset is too large for fast-enough handling of these queries by SQL-compatible DB.

It looks like PostgreSQL can do the job. If you're looking for columnar-DB for lighting-fast OLAP queries (aggregations) you can check open-source ClickHouse.

Finally, Dremio is not the only BI tool that can work with ElasticSearch (or PostgreSQL, ClickHouse etc). Some BI tool allows you to use 'painless' scripts for dimensions/measures and you can calculate a lot of things directly in ES queries.

Upvotes: 4

Related Questions