Advice on remote monitoring strategy

Question

I need to monitor remote software I have running in several POS (point of sale) stations.

The scenario:

• POS stations are distributed across the country
• POS stations have internet connections
• POS stations have no PUBLIC IP address (they're behind NAT firewalls)
• POS is running an agent (is a Python agent, but I don't think it matters) that could be used to send monitoring info.

What I need:

• I need to know if the agent is running
• I need to know size of agent's work queue

What I've thought of (but I think is quite old school):

• Do a sort of HTTP PING to an CUSTOM/GRAFANA/NAGIOS endpoint
• Use XMPP (I like {but it's not essential} the idea of having a persistent TCP/IP socket open to my server infrastructure)
• WebSockets (using it similarly to XMPP)

====

What are your thoughts and experiences about this kind of software issues?

Answer 1

At my place of work, we have effectively used telegraf and the TICK stack in general for this sort of PUSH based metrics/monitoring.

Whether you use telegraf or not, the key here would be to have regular PING like messages coming from each remote node. You can look for the absence of two consecutive messages from any particular node as evidence of a problem.

Depending on how important this monitoring is, you may want to consider redundant receivers of metrics on two different cloud vendors for redundancy.