Monit EXEC not working when monitored process dies

Question

using Monit 5.15 on FreeBSD 10.2:

set daemon  5
set logfile syslog
set pidfile /var/run/monit.pid
set idfile /var/.monit.id
set statefile /var/.monit.state
set alert x@y.z
set mailserver localhost
set httpd port 2812 and
     use address 192.168.40.72
     allow 192.168.20.0/24
     allow admin:monit

check process haproxy with pidfile /var/run/haproxy.pid
     if failed host 192.168.40.72 port 9090 type tcp
       then exec "/bin/sh -c '/bin/echo `/bin/date` >> /tmp/monit.test'"

When i run monit with -vI and i kill haproxy, i have the following output:

Adding net allow '192.168.20.0/24'
Adding credentials for user 'admin'
Runtime constants:
 Control file       = /usr/local/etc/monitrc
 Log file           = syslog
 Pid file           = /var/run/monit.pid
 Id file            = /var/.monit.id
 State file         = /var/.monit.state
 Debug              = True
 Log                = True
 Use syslog         = True
 Is Daemon          = True
 Use process engine = True
 Poll time          = 5 seconds with start delay 0 seconds
 Expect buffer      = 256 bytes
 Mail server(s)     = localhost:25 with timeout 30 seconds
 Mail from          = (not defined)
 Mail subject       = (not defined)
 Mail message       = (not defined)
 Start monit httpd  = True
 httpd bind address = 192.168.40.72
 httpd portnumber   = 2812
 httpd ssl          = Disabled
 httpd signature    = Enabled
 httpd auth. style  = Basic Authentication and Host/Net allow list
 Alert mail to      = root@localhost
   Alert on         = All events

The service list contains the following entries:

Process Name          = haproxy
 Pid file             = /var/run/haproxy.pid
 Monitoring mode      = active
 Existence            = if does not exist then restart
 Port                 = if failed [192.168.40.72]:9090 type TCP/IP protocol DEFAULT with timeout 5 seconds then exec '/bin/sh -c /bin/echo `/bin/date` >> /tmp/monit.test'

System Name           = appsrv01
 Monitoring mode      = active

-------------------------------------------------------------------------------
pidfile '/var/run/monit.pid' does not exist
Starting Monit 5.15 daemon with http interface at [192.168.40.72]:2812
Starting Monit HTTP server at [192.168.40.72]:2812
Monit HTTP server started
'appsrv01' Monit 5.15 started
Sending Monit instance changed notification to root@localhost
'haproxy' process is running with pid 42999
'haproxy' zombie check succeeded
'haproxy' succeeded testing protocol [DEFAULT] at [192.168.40.72]:9090 [TCP/IP]
'haproxy' connection succeeded to [192.168.40.72]:9090 [TCP/IP]
'haproxy' process is running with pid 42999
'haproxy' zombie check succeeded
'haproxy' succeeded testing protocol [DEFAULT] at [192.168.40.72]:9090 [TCP/IP]
'haproxy' connection succeeded to [192.168.40.72]:9090 [TCP/IP]
'haproxy' process is running with pid 42999
'haproxy' zombie check succeeded
'haproxy' succeeded testing protocol [DEFAULT] at [192.168.40.72]:9090 [TCP/IP]
'haproxy' connection succeeded to [192.168.40.72]:9090 [TCP/IP]
'haproxy' process test failed [pid=42999] -- No such process
'haproxy' process is not running
Sending Does not exist notification to root@localhost
'haproxy' trying to restart
'haproxy' stop skipped -- method not defined
'haproxy' start method not defined
'haproxy' monitoring enabled
'haproxy' process test failed [pid=42999] -- No such process
'haproxy' process is not running
'haproxy' trying to restart
'haproxy' stop skipped -- method not defined
'haproxy' start method not defined
'haproxy' monitoring enabled
^CShutting down Monit HTTP server
Monit HTTP server stopped
Monit daemon with pid [48685] stopped
'appsrv01' Monit 5.15 stopped
Sending Monit instance changed notification to root@localhost

The EXEC Line never gets executed, i dont see any new lines in /tmp/monit.test

If i change the checked Port from 9090 to some invalid port, lets say 9190 and start monit (haproxy is running !), i see:

Starting Monit 5.15 daemon with http interface at [192.168.40.72]:2812
Starting Monit HTTP server at [192.168.40.72]:2812
Monit HTTP server started
'appsrv01' Monit 5.15 started
Sending Monit instance changed notification to root@localhost
'haproxy' process is running with pid 50703
'haproxy' zombie check succeeded
Socket test failed for [192.168.40.72]:9190 -- Connection refused
'haproxy' failed protocol test [DEFAULT] at [192.168.40.72]:9190 [TCP/IP] -- Connection refused
Sending Connection failed notification to root@localhost
'haproxy' exec: /bin/sh
'haproxy' process is running with pid 50703
'haproxy' zombie check succeeded
Socket test failed for [192.168.40.72]:9190 -- Connection refused
'haproxy' failed protocol test [DEFAULT] at [192.168.40.72]:9190 [TCP/IP] -- Connection refused
'haproxy' exec: /bin/sh

Why does the EXEC Line works here but not when i kill -9 haproxy ? What i'm trying to do is get monit to run the exec in case of a haproxy failure. the exec line will then contain a command to switch the CARP IP to another host. haproxy itself is monitored using zabbix, so the NOC can investigate the cause of the failure later.

Answer 1

When you kill -9 haproxy you're killing the daemon. So when monit performs this "check process" block, it's detecting that the process isn't there and restarting the process. It doesn't perform the check on that port because it sees the process isn't there.

It works when you give it an invalid port because the process is still alive. When it performs the port check it will fail that and run the script.

You should add an additional line to this check block that says

check process haproxy with pidfile /var/run/haproxy.pid
     if failed host 192.168.40.72 port 9090 type tcp 
         then exec "/bin/sh -c '/bin/echo `/bin/date` >> /tmp/monit.test'"
     if restarted then exec "/bin/sh -c '/bin/echo `/bin/date` >>/tmp/monit.test'"

This should run the shell commands on both a restart AND a failed host.