Architecture for Logstash and how to deploy

Question

I am tasked to set up an Elastic stack on OpenShift. I have limited experience in both. I planned to use the Elastic OpenShift operator (https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-openshift.html). What made me think, why is there no Logstash included in the said operator?

• Is Logstash not mentioned in the operator as it should be installed on each node?
• If Logstash is supposed to be installed on OpenShift, is the idea to use the Helm charts? (https://github.com/elastic/helm-charts/tree/main/logstash)

I understood the general Elastic architecture in such a way:

• One or multiple Elasticsearch nodes create a single cluster which stores all data. In addition, Elasticsearch can be run in containers.
• One or multiple Kibana nodes connect to the Elasticsearch cluster to read the data. Kibana can be run in containers.
• Filebeat nodes are installed for each application, node or whatever else has to be sent to the Elasticserach cluster. Filebeat is installed on each instance.
• Logstash collects data from sources which Filebeat can not read. Logstash also has the ability to modify data using pipelines. Where does one run Logstash nodes and or clusters?

I hope someone can answer some of my many questions or direct me to a resource.

Thanks

Answer 1

Logstash is a log aggregator. It collects & filters logs and sends them to ElasticSearch database. OpenShift has it's own log agregator: https://docs.openshift.com/container-platform/4.10/logging/cluster-logging-deploying.html

Therefore you can forward container logs using this directly to Elasticsearch. You can then install Filebeats to forward logs written to PVCs to Elasticsearch as well.

You can easily install logstash, elasticsearch & kibana on OpenShift using helm:

• https://artifacthub.io/packages/helm/elastic/elasticsearch
• https://artifacthub.io/packages/helm/elastic/logstash
• https://artifacthub.io/packages/helm/elastic/kibana