CODEWITHSUNDEEP
amazon-web-serviceselasticsearchloggingaws-fargateelk
KasperF
KasperF

Reputation: 342

Best approach for sending logs from ECS Fargate into Elasticsearch

We have a setup with multiple containers running NodeJS services(node:11-alpine docker image) deployed in AWS ECS Fargate.

We already have a running ElasticSearch instance collecting logs from non-Fargate application. I would like to pass the logs from the Fargate containers into this ElasticSearch instance, but I have a hard time to figure out what is the best approach.

1) It seem one way is to stream the logs from Cloudwatch --> Lambda --> ElasticSearch. It seem a bit overkill - isn't there another way to do this?

2) I was hoping i could run a Logstash docker instance that could collect the logs from the containers but I am not sure if this is possible when running Fargate?

3) Should I install something like FileBeat on each container and let that send the logs?

Any help is appreciated.

Upvotes: 12

Views: 11131

Answers (1)

Abdullah Khawer
Abdullah Khawer

Reputation: 5778

  1. It seems one way is to stream the logs from Cloudwatch --> Lambda --> ElasticSearch. It seem a bit overkill - isn't there another way to do this?

If you're looking for an AWS-based managed solution, that is one of the ways. You don't really need to write a Lambda function, AWS does it for you. Although, you have to bear the cost of Lambda and CloudWatch.

There is another solution that is recommended by AWS and that is the use of fluent-bit as a sidecar container to export logs directly to Elasticsearch/OpenSearch from other containers running within a service. Using this solution, you save money by not using AWS CloudWatch. This solution also provides better results with regard to the loss of logs upon failure.

  1. I was hoping I could run a Logstash docker instance that could collect the logs from the containers but I am not sure if this is possible when running Fargate?

Yes, that is possible if you run that container along with the other container.

  1. Should I install something like FileBeat on each container and let that send the logs?

You can use Fluent Bit, Filebeat, Fluentd, Functionbeat, or Logstash as you like.

Note: If you're thinking of running your own logs exporter container like Logstash, Fluent Bit, etc, don't enable CloudWatch logging to save money as you're not going to use that.

Upvotes: 10

Related Questions