CODEWITHSUNDEEP
javarundeck
lingz
lingz

Reputation: 321

Rundeck in bad state, restarting spawns multiple broken instances

I recently encountered a situation with Rundeck where service logs indicated that Rundeck was still functional, but the web gui was down, and lsof -i :4443 indicated that nothing was listening on Rundeck's web port. The rundeck commandline was also down, as all the rd commands (e.g. rd-queue) were not returning.

rundeckd restart (alternatively rundeckd stop; rundeckd start gave correct output, but only spawned more processes running the rundeck jar. The final solution was to force kill all of these processes and start rundeck via the init script.

  1. Is there a more sophisticated way to check if Rundeck is still up aside from checking logs and rundeckd status? Status said it was up and running, which it most certainly was not.

  2. What might cause Rundeck to enter this state? Is it possible for rundeck to still be functional / executing jobs and merely the web UI is down? Is it possible to restart or fix the web UI only without restarting rundeck and thus killing all running jobs entirely?

Upvotes: 0

Views: 532

Answers (1)

Alex-SF
Alex-SF

Reputation: 76

The rundeckd status command is the best way to check status. The service.log can also contain possible errors that might include out of memory or database connection errors.

The rd-* commands like rd-queue are clients to the Rundeck server process so will not function if the daemon is down.

It is possible that when doing the rundeck stop/start/restart action was forking duplicate processes which will result in errors in the service.log about socket bind errors.

Upvotes: 2

Related Questions