CODEWITHSUNDEEP
databaseelasticsearchdatabase-designarchitectureelastic-stack
MaxThom
MaxThom

Reputation: 1421

Elastic Search - MultiLog design

we are trying to implement a new log system for our IoT device, different applications in the cloud (api, spa, etc). We are trying to design the "Schema" to be the most efficient as possible and we feel there are many good solutions, but it's hard to select one.

Here is a general structure : under the devices node we have our 3 different kinds of IoT devices and similar for infra : different applications and more. enter image description here So we were thinking of creating one index for each blue circle and create a hierarchical naming with our indexes so we can take advantage of the wildcard when execute search.

For example :

And for mapping, we have different log type in each index and should we map only the common field or everything ? Should we map common field and let the dynamic mapping for the logs type specifics?

Please share your opinion and tips if you have !

Ty.

Upvotes: 0

Views: 42

Answers (1)

xeraa
xeraa

Reputation: 10859

I would generally map everything to ECS, since Kibana knows the meaning of many fields and it aligns with other inputs.

How much data do you have and how different are your fields? If you don't have too much data (every shard should have >10GB — manage with rollover / ILM ideally) and less than 100 fields in total, I would go for a single index and add a field with with the different names, so you can easily filter on that. Though different retention lengths of the data would favor multiple indices, so you will have to pick the right tradeoffs for your system.

Upvotes: 1

Related Questions