CODEWITHSUNDEEP
pythonmongodbpostgresqlubuntumonitoring
SarthakDev
SarthakDev

Reputation: 183

Monitoring tool for distributed micro-services

So I have a set of distributed micro-services, for eg. Elasticsearch, MongoDB, Postgres, a Flask application, an Apache Kafka cluster etc. I want to build a monitoring tool where I can have a dashboard for all kinds of stats around them. For eg, run-time, CPU %, number of processes et al. These and the obvious current status which would tell me if the service is running or not.

Never made a tool like this before, so could do with some help. Is there something already made for this? If not, what are the good practices for making such a thing?

Upvotes: 3

Views: 1221

Answers (3)

Olaf Molenveld
Olaf Molenveld

Reputation: 11

You might take a look at www.vamp.io, an opensource JVM-based framework to manage container and (micro)services based architectures.

It also includes monitoring and a GUI with dashboard functionalities. Vamp uses ElasticSearch to store the metrics flowing through the system, so you can also easily create your own custom Kibana dashboard if you want. There is a powerful metrics API you can use: http://vamp.io/documentation/using-vamp/metrics-and-events/

We're moving into beta-stage now, but Vamp is already used in production at scale at some SaaS-providers.

Disclaimer: i am one of the founders of Magnetic.io, the company behind Vamp.

Upvotes: 1

Steve Barnes
Steve Barnes

Reputation: 28370

For python you can use psutil which has a bunch of facilities for monitoring the hardware, etc.

  • cpu_times
  • virtual_memory
  • swap_memory
  • Process Monitoring
  • Network Monitoring,
  • etc.

There is also glances which can be run in client/server web mode. Snapshot of glances running on my machine in a web browser.

Glances in a browser

Already build from psutil is GRR Rapid Response which is an incident response framework focused on remote live forensics and includes remote detailed monitoring of client CPU, memory, IO usage and self-imposed limits..

Upvotes: -1

meson10
meson10

Reputation: 1954

Look at Sensu. Sensu is a pretty advanced monitoring tool. Think of it like distributed nagios. So you install a Sensu agent on each machine which has a large no. Of plugins available for each of the processes you mentioned above. You can also write your own plugins in a language of your choice as long as they are valid executables. Bash, perl, Python, ruby anything would work. Agents publish data to a rabbitmQ, where on data is read by the Sensu server. For dashboards you can simply install uchiwa, which is probably the best Sensu dashboard out there. This allows you to do stuff beyond just monitoring because Sensu server requests updates from all Sensu agents via rabbitmQ. So you can use it like your own event trigger on these machines. Like you could remotely start and shutdown processes too.

Another noteworthy tool is Riemann, http://riemann.io/.

While Sensu is more about sending data via agents, Riemann is all about stream processing at receiving end. So you could have Sensu agents and a Riemann receiver. With Riemann you can build really advanced rules of metric alerts.

Then there's Icinga. Which is also advanced nagios but not as comprehensive as Sensu is.

Lastly if you don't want to go through the process of setting up and maintaining all this, you can just use a hosted service like datadog. But there you would pay for every host you monitor.

Upvotes: 2

Related Questions