Reputation: 1104
I am new to AWS and was experimenting with some of its services. I managed to create a python application running inside an EC2 instance. The application creates a log file with the analysis data. I want to connect this log file with AWS's Elasticsearch and Kibana service to begin running analytics on it. Can someone point me to the best way of streaming my EC2 app's logs to AWS elasticsearch service.
Upvotes: 0
Views: 1339
Reputation: 60164
You have multiple options to deal with this problem. in case of AWS
Lamda will push logs to ELK
But I will go with the below approach as it will not need Lambda and log-group and the logs will send to ELK directly.
Filebeat is part of the Elastic Stack, meaning it works seamlessly with Logstash, Elasticsearch, and Kibana. Whether you want to transform or enrich your logs and files with Logstash, fiddle with some analytics in Elasticsearch, or build and share dashboards in Kibana, Filebeat makes it easy to ship your data to where it matters most.
All you need to sepcify application log files.
paths:
- /app/log/*.log
- /app/log/*/*.log
Logagent is a modern, open-source, light-weight log shipper. It is like Filebeat and Logstash in one, without the JVM memory > footprint. It comes with out of the box and extensible log parsing, > on-disk buffering, secure transport, and bulk indexing to > Elasticsearch, Sematext Logs, and other destinations. Its low memory > footprint and low CPU overhead make it suitable for deploying on edge > nodes and devices, while its ability to parse and structure logs makes > it a great Logstash alternative.
sudo npm i -g @sematext/logagent
shipping-data-to-aws-elasticsearch-with-logagent
Upvotes: 1