CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2elk
abhinav singh
abhinav singh

Reputation: 1104

Using AWS ELK stack for capturing logs

I am new to AWS and was experimenting with some of its services. I managed to create a python application running inside an EC2 instance. The application creates a log file with the analysis data. I want to connect this log file with AWS's Elasticsearch and Kibana service to begin running analytics on it. Can someone point me to the best way of streaming my EC2 app's logs to AWS elasticsearch service.

Upvotes: 0

Views: 1339

Answers (1)

Adiii
Adiii

Reputation: 60164

You have multiple options to deal with this problem. in case of AWS

  • Install aws cloud watch log agent
  • Start log-agent with log file
  • Stream cloud watch log to lambda

  • Lamda will push logs to ELK

    enter image description here

But I will go with the below approach as it will not need Lambda and log-group and the logs will send to ELK directly.

  • Filebeat
  • Logagent (node based pacakge)

Filebeat is part of the Elastic Stack, meaning it works seamlessly with Logstash, Elasticsearch, and Kibana. Whether you want to transform or enrich your logs and files with Logstash, fiddle with some analytics in Elasticsearch, or build and share dashboards in Kibana, Filebeat makes it easy to ship your data to where it matters most.

enter image description here

All you need to sepcify application log files.

paths:
- /app/log/*.log
- /app/log/*/*.log

Logagent is a modern, open-source, light-weight log shipper. It is like Filebeat and Logstash in one, without the JVM memory > footprint. It comes with out of the box and extensible log parsing, > on-disk buffering, secure transport, and bulk indexing to > Elasticsearch, Sematext Logs, and other destinations. Its low memory > footprint and low CPU overhead make it suitable for deploying on edge > nodes and devices, while its ability to parse and structure logs makes > it a great Logstash alternative.

sudo npm i -g @sematext/logagent

enter image description here shipping-data-to-aws-elasticsearch-with-logagent

Upvotes: 1

Related Questions