Implementing cloud watch like metric aggregator for real time monitoring of applications

Question

I want to design something like amazon cloud watch for real time monitoring of applications I own and I run in my own cluster of machines (not AWS cluster).

Basic requirement:

• Ability to publish time and count metric from applications and view them in a Kibana like dashboard.
• Seamless support to publish metrics from applications written in Python, Java and Node.js.

Possible approach that I think will work but I need suggestions on:

• I am thinking of writing a daemon process that will run on all the hosts in my cluster.
• Applications will publish metrics to a file system in a format that this daemon understands. Not sure how this contract will be created for applications written in different languages?
• Daemon will pull this information say every 5 seconds and publish it to the ELK. Not sure how Daemon will distinguish one application from the other. I guess I will have to create some metric registry service that will register applications. I am not sure what all aspects metric registry will be taking care off.

Pointer on this are really appreciated.

Answer 1

For your basic requirements.

• Create a simple java(or your choice of language) application that will send data to elastic search, something like ElasticSearchClientSender. Check Jest, we use that to send data to our ES. Setup your kibana with the the correct alias. Creating it as a module may not be necessary in your case, as you may just create a snippet and plug it to the application you want its data published to ES.

If you still need the cloudwatch metric for other monitoring purposes, either you could try below options

• AWS Java SDK got you covered. EDIT they also have one for python, nodejs and others(we mostly use python)
• Or in our case, we have setup each of our module to have its own Cloudwatch Log Groups, so our elastic search sender module has it own log groups. Overview of Cloudwatch logs. When you have setup the Cloudwatch Log Groups for your module, decide what keyword you want to indicate what you want to show in the Metric and just log the data as you like, refer to these these samples

Another Approach

Based on your listed other approaches, maybe it's a worth a try looking for AWS Lambda , instead of creating a module that sends data with a snippet of code, In lambda, you will still have a chunk of code as a standalone application that you will setup on how it will be triggered depending on your setup either cronjob style or some aws services with Cloudwatch Events