CODEWITHSUNDEEP
rundeck
randomScott
randomScott

Reputation: 316

Incorrect failure notification from Rundeck during fall time change

Last night was "fall back" time change for most locations in the US. I woke up this morning to find dozens of job failure notifications. Almost all of them though were incorrect: the jobs showed as having completed normally, yet Rundeck sent a failure notification for it.

Interestingly, this happened in two completely separate Rundeck installations (v2.10.8-1 and v3.1.2-20190927). The commonality is that they're both on CentOS 7 (separate servers). They're both using MariaDB, although different versions of MariaDB.

The failure emails for the jobs that finished successfully showed a negative time in the "Scheduled after" line:

#1,811,391 
by admin Scheduled after 59m at 1:19 AM 
• Scheduled after -33s - View Output » 
• Download Output 
Execution 
User:   admin
Time:   59m
Started:    in 59m  2019-11-03 01:19:01.0
Finished:   1s ago  Sun Nov 03 01:19:28 EDT 2019
    Executions  Success rate    Average duration 
    100%    -45s

That job actually ran in 27s at 01:19 EDT (the first 1am hour, it is now EST). Looking at the email headers, I believe I got the message at 1:19 EST, an hour after the job ran.

So that would seem to imply to me that it's just a notification problem (somehow).

But there were a couple of jobs that were following other job executions that failed as well, apparently because the successfully finished job returned a RC 2. I'm not sure what to make of this.

We've been running Rundeck for a few years now, this is the first I remember seeing this problem. Of course my memory may be faulty--maybe we did see it previously, only there were fewer jobs affected or some such.

The fact that it impacted two different versions of Rundeck on two different servers implies either it's a fundamental issue with Rundeck that's been around for a while or it is something else in the operating system that's somehow causing problems for Rundeck. (Although time change isn't new, so that would seem to be somewhat surprising too.)

Any thoughts about what might have gone on (and how to prevent it next year, short of the obvious run on UTC) would be appreciated.

Upvotes: 0

Views: 268

Answers (1)

MegaDrive68k
MegaDrive68k

Reputation: 4325

You can define specific Timezone in Rundeck, check this and this.

Upvotes: 0

Related Questions