CODEWITHSUNDEEP
rabbitmqrabbitmq-shovel
nemo
nemo

Reputation: 1584

RabbitMQ Shovel Stuck in 'Terminated' Status

We have an issue where once in a while, a dynamic shovel (created via the HTTP API: /api/parameters/shovel/) with src-delete-after set to queue-length finishes and then instead of being deleted, gets stuck in a terminated status.

Subsequent attempts to delete the shovel by any of the following methods are unsuccessful:

  1. Posting a DELETE to /api/parameters/shovel/
  2. rabbitmqctl delete_shovel
  3. rabbitmqctl clear_parameter -p <vhost> shovel <shovel_name>

The shovel doesn't even appear in the 'Shovel Management' section of the RabbitMQ admin UI.

The only way we could get rid of that stuck shovel is by restarting RabbitMQ.

Is anyone else having this issue? If so, how do we clear the shovel without having to restart the cluster? Also, is it possible to prevent this from happening via configuration?

Thanks!

PS:

  1. RabbitMQ version: 3.4.4
  2. Running a 2 node cluster (will be making it a 3 node cluster shortly due to obvious issue we could face in the case of a network partition).

RabbitMQ Shovel Stuck in 'Terminated' Status

Upvotes: 1

Views: 3315

Answers (2)

Luke Bakken
Luke Bakken

Reputation: 9657

You are using a very, very old version of RabbitMQ. Please upgrade to the latest version (3.7.6) and be sure to use Erlang 20.3.X (not 21).

Upvotes: 2

Lee Grissom
Lee Grissom

Reputation: 9953

We're using RMQ 3.7.13, Erlang 21.3.1.

One possible way the problem happens:

  • 3 node HA cluster
  • restart one of the nodes (/etc/init.d/rabbitmq-server restart)
  • old shovels AND old queues get resurrected somehow

The shovels can't be deleted using any of the ways mentioned in the question. The only way I was able to get the shovels removed was to disable the shovel plugin on all 3 nodes in the cluster, then re-enable each plugin on each node like so:

rabbitmq-plugins disable rabbit_shovel
rabbitmq-plugins enable rabbit_shovel
rabbitmq-plugins enable rabbit_shovel_management

As far as the old queues getting resurrected (happens randomly not touching anything... I call them "zombie" queues), this problem happens about once a month, so I created PostMan scripts to delete the resurrected queues. This has been a problem for years. We upgraded RMQ in hopes that fixes the issue... but it doesn't. Perhaps Quorum queues are a more robust solution? If I had more time to investigate/experiment, I would, but I'm buried up to my eyeballs with higher priorities.

Upvotes: 2

Related Questions