RabbitMQ virtual host error when starting service

Question

I've had a RabbitMQ server running for months. This morning I was unable to connect to it, my applications was timing out and the Management client was unresponsive. Rebooted the machine. Applications are still timing out. I'm able to login to the Management client but I see this message:

Virtual host / experienced an error on node rabbit@MQT01 and may be inaccessible

All my queues are there but can't see any exchanges.

I hope someone can help me figure out what going on. I've looked at the logs but can't find any good hint.

Here a part of the log:

2018-09-11 09:39:42 =ERROR REPORT====
** Generic server <0.281.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.36.122888644>}}
** When Server state == undefined
** Reason for termination == 
** {function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]}
2018-09-11 09:39:42 =CRASH REPORT====
  crasher:
    initial call: worker_pool_worker:init/1
    pid: <0.281.0>
    registered_name: []
    exception exit: {{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]},[{gen_server2,terminate,3,[{file,"src/gen_server2.erl"},{line,1161}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
    ancestors: [worker_pool_sup,rabbit_sup,<0.262.0>]
    message_queue_len: 0
    messages: []
    links: [<0.276.0>,<0.336.0>,#Port<0.31196>]
    dictionary: [{fhc_age_tree,{1,{{10352640,#Ref<0.1077581647.1695285251.67028>},true,nil,nil}}},{worker_pool_worker,true},{rand_seed,{#{jump => #Fun<rand.16.15449617>,max => 288230376151711743,next => #Fun<rand.15.15449617>,type => exsplus},[257570830250844431|246837015578235662]}},{worker_pool_name,worker_pool},{{"c:/Users/dfpsb/AppData/Roaming/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/9GD33C2I2PKZ7A8QHZ4MWWCKE/journal.jif",fhc_file},{file,1,true}},{{#Ref<0.1077581647.1695285251.67028>,fhc_handle},{handle,{file_descriptor,prim_file,{#Port<0.31196>,1808}},#Ref<0.1077581647.1695285251.67028>,240,false,0,infinity,[],<<>>,0,0,0,0,0,false,"c:/Users/dfpsb/AppData/Roaming/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/9GD33C2I2PKZ7A8QHZ4MWWCKE/journal.jif",[write,binary,raw,read],[{write_buffer,infinity}],true,true,10352640}}]
    trap_exit: false
    status: running
    heap_size: 10958
    stack_size: 27
    reductions: 104391
  neighbours:
    neighbour: [{pid,<0.279.0>},{registered_name,[]},{initial_call,{worker_pool_worker,init,['Argument__1']}},{current_function,{gen,do_call,4}},{ancestors,[worker_pool_sup,rabbit_sup,<0.262.0>]},{message_queue_len,0},{links,[<0.276.0>,<0.336.0>]},{trap_exit,false},{status,waiting},{heap_size,4185},{stack_size,42},{reductions,21548},{current_stacktrace,[{gen,do_call,4,[{file,"gen.erl"},{line,169}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,210}]},{file,call,2,[{file,"file.erl"},{line,1499}]},{rabbit_queue_index,get_journal_handle,1,[{file,"src/rabbit_queue_index.erl"},{line,881}]},{rabbit_queue_index,load_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,894}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,904}]},{rabbit_queue_index,scan_queue_segments,3,[{file,"src/rabbit_queue_index.erl"},{line,724}]},{rabbit_queue_index,queue_index_walker_reader,2,[{file,"src/rabbit_queue_index.erl"},{line,712}]}]}]
    neighbour: [{pid,<0.278.0>},{registered_name,[]},{initial_call,{worker_pool_worker,init,['Argument__1']}},{current_function,{gen,do_call,4}},{ancestors,[worker_pool_sup,rabbit_sup,<0.262.0>]},{message_queue_len,0},{links,[<0.276.0>,<0.336.0>,#Port<0.31157>]},{trap_exit,false},{status,waiting},{heap_size,6772},{stack_size,102},{reductions,129623},{current_stacktrace,[{gen,do_call,4,[{file,"gen.erl"},{line,169}]},{gen_server2,call,3,[{file,"src/gen_server2.erl"},{line,323}]},{array,sparse_foldr_3,6,[{file,"array.erl"},{line,1848}]},{array,sparse_foldr_2,8,[{file,"array.erl"},{line,1837}]},{lists,foldr,3,[{file,"lists.erl"},{line,1276}]},{rabbit_queue_index,scan_queue_segments,3,[{file,"src/rabbit_queue_index.erl"},{line,725}]},{rabbit_queue_index,queue_index_walker_reader,2,[{file,"src/rabbit_queue_index.erl"},{line,712}]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_index.erl"},{line,694}]}]}]
    neighbour: [{pid,<0.280.0>},{registered_name,[]},{initial_call,{worker_pool_worker,init,['Argument__1']}},{current_function,{array,set_1,4}},{ancestors,[worker_pool_sup,rabbit_sup,<0.262.0>]},{message_queue_len,0},{links,[<0.276.0>,<0.336.0>,#Port<0.31170>]},{trap_exit,false},{status,runnable},{heap_size,121536},{stack_size,44},{reductions,122988},{current_stacktrace,[{array,set_1,4,[{file,"array.erl"},{line,590}]},{array,set_1,4,[{file,"array.erl"},{line,592}]},{array,set_1,4,[{file,"array.erl"},{line,592}]},{array,set,3,[{file,"array.erl"},{line,574}]},{rabbit_queue_index,parse_segment_publish_entry,5,[{file,"src/rabbit_queue_index.erl"},{line,1135}]},{rabbit_queue_index,segment_entries_foldr,3,[{file,"src/rabbit_queue_index.erl"},{line,1091}]},{lists,foldr,3,[{file,"lists.erl"},{line,1276}]},{rabbit_queue_index,scan_queue_segments,3,[{file,"src/rabbit_queue_index.erl"},{line,725}]}]}]
    neighbour: [{pid,<0.336.0>},{registered_name,[]},{initial_call,{gatherer,init,['Argument__1']}},{current_function,{gen_server2,process_next_msg,1}},{ancestors,[<0.332.0>,<0.324.0>,<0.323.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.262.0>]},{message_queue_len,2},{links,[<0.280.0>,<0.332.0>,<0.281.0>,<0.278.0>,<0.279.0>]},{trap_exit,false},{status,runnable},{heap_size,987},{stack_size,8},{reductions,73223},{current_stacktrace,[{gen_server2,process_next_msg,1,[{file,"src/gen_server2.erl"},{line,666}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}]
2018-09-11 09:39:42 =CRASH REPORT====
  crasher:
    initial call: rabbit_msg_store:init/1
    pid: <0.332.0>
    registered_name: []
    exception exit: {{{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]},{gen_server2,call,[<0.336.0>,out,infinity]}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,589}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
    ancestors: [<0.324.0>,<0.323.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.262.0>]
    message_queue_len: 1
    messages: [{'EXIT',<0.336.0>,{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]}}]
    links: [<0.335.0>,<0.324.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 57377
  neighbours:
    neighbour: [{pid,<0.335.0>},{registered_name,[]},{initial_call,{rabbit_msg_store_gc,init,['Argument__1']}},{current_function,{gen_server2,process_next_msg,1}},{ancestors,[<0.332.0>,<0.324.0>,<0.323.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.262.0>]},{message_queue_len,0},{links,[<0.332.0>]},{trap_exit,false},{status,waiting},{heap_size,987},{stack_size,8},{reductions,174},{current_stacktrace,[{gen_server2,process_next_msg,1,[{file,"src/gen_server2.erl"},{line,666}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}]
2018-09-11 09:39:42 =SUPERVISOR REPORT====
     Supervisor: {local,worker_pool_sup}
     Context:    child_terminated
     Reason:     {function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]}
     Offender:   [{pid,<0.281.0>},{id,4},{mfargs,{worker_pool_worker,start_link,[worker_pool]}},{restart_type,transient},{shutdown,4294967295},{child_type,worker}]

2018-09-11 09:39:42 =CRASH REPORT====
  crasher:
initial call: rabbit_vhost_process:init/1
pid: <0.325.0>
registered_name: []
exception exit: {{error,{{{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]},{gen_server2,call,[<0.336.0>,out,infinity]}},{child,undefined,msg_store_persistent,{rabbit_msg_store,start_link,[msg_store_persistent,"c:/Users/dfpsb/AppData/Roaming/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",[],{#Fun<rabbit_queue_index.2.122888644>,{start,[{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Test1.InterchangeFragtbrevEnvelope.RET">>},{resource,<<"/">>,queue,<<"Test2.DfLoggingEvent.Debug">>},{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Paw.DfLoggingEvent.Debug">>},{resource,<<"/">>,queue,<<"DevUnitTest.TruckLoadingEnvelope.UnitTest">>},{resource,<<"/">>,queue,<<"Test1.InterchangeFragtbrevEnvelope.RET_error">>},{resource,<<"/">>,queue,<<"Paw.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"DevUnitTest.TestMsg.UnitTest_error">>},{resource,<<"/">>,queue,<<"DevUnitTest.TestMsg.UnitTest">>},{resource,<<"/">>,queue,<<"Paw.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Test2.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Paw.InterchangeFragtbrevEnvelope.TurPlan_error">>},{resource,<<"/">>,queue,<<"Paw.TruckLoadingEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Test2.InterchangeFragtbrevEnvelope.TurPlan_error">>},{resource,<<"/">>,queue,<<"Test2.TruckLoadingEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Paw.DfLoggingEvent.Warning">>},{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeFragtbrevEnvelope.RET">>},{resource,<<"/">>,queue,<<"Test2.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Test2.DfLoggingEvent.Warning">>}]}}]},transient,30000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,581}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
ancestors: [<0.323.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.262.0>]
message_queue_len: 0
messages: []
links: [<0.323.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 10958
stack_size: 27
reductions: 63314
  neighbours:
2018-09-11 09:39:42 =SUPERVISOR REPORT====
 Supervisor: {<0.323.0>,rabbit_vhost_sup_wrapper}
 Context:    start_error
 Reason:     {error,{{{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]},{gen_server2,call,[<0.336.0>,out,infinity]}},{child,undefined,msg_store_persistent,{rabbit_msg_store,start_link,[msg_store_persistent,"c:/Users/dfpsb/AppData/Roaming/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",[],{#Fun<rabbit_queue_index.2.122888644>,{start,[{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Test1.InterchangeFragtbrevEnvelope.RET">>},{resource,<<"/">>,queue,<<"Test2.DfLoggingEvent.Debug">>},{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Paw.DfLoggingEvent.Debug">>},{resource,<<"/">>,queue,<<"DevUnitTest.TruckLoadingEnvelope.UnitTest">>},{resource,<<"/">>,queue,<<"Test1.InterchangeFragtbrevEnvelope.RET_error">>},{resource,<<"/">>,queue,<<"Paw.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"DevUnitTest.TestMsg.UnitTest_error">>},{resource,<<"/">>,queue,<<"DevUnitTest.TestMsg.UnitTest">>},{resource,<<"/">>,queue,<<"Paw.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Test2.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Paw.InterchangeFragtbrevEnvelope.TurPlan_error">>},{resource,<<"/">>,queue,<<"Paw.TruckLoadingEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Test2.InterchangeFragtbrevEnvelope.TurPlan_error">>},{resource,<<"/">>,queue,<<"Test2.TruckLoadingEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Paw.DfLoggingEvent.Warning">>},{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeFragtbrevEnvelope.RET">>},{resource,<<"/">>,queue,<<"Test2.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Test2.DfLoggingEvent.Warning">>}]}}]},transient,30000,worker,[rabbit_msg_store]}}}
 Offender:   [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"/">>]}},{restart_type,permanent},{shutdown,30000},{child_type,worker}]

Answer 1

In my case, there was a problem with corrupted data on a disk. What I did:

1. Logged in to the host reporting issue

2. Stopped the RabbitMQ application and resetting it

   sudo rabbitmqctl start_app
   sudo rabbitmqctl reset
   

3. Removed directory /var/lib/rabbitmq/mnesia/{vhost_name}

4. Removed faulty node from the cluster by logging into one of the existing nodes in the RabbitMQ cluster and running the given command:

   sudo rabbitmqctl forget_cluster_node {vhost_name}@{faulty_node_name}
   

5. On the faulty node, re-join the cluster

   sudo rabbitmqctl join_cluster {vhost_name}@{any_working_cluster_node}
   

6. Enable the RabbitMQ app

   sudo rabbitmqctl start_app
   

Answer 2

What worked for me:

use sudo to run rabbitmq

sudo rabbitmqctl stop

sudo rabbitmq-server -detached  

Answer 3

I figured out what was going on. Someone on my team (me) was creating an unprecedented amount of connections to RabbitMQ. For each connection a file handler is created to one or more files, not sure which. The OS (Windows in my case) has a file handler limit, not sure what the limit is, but when reached, an error is thrown. This corrupted the virtual host (/) and I had to delete it and create it again. Good thing this wasn't production, because then all the messages would've been gone.

Edit (June 2020):

There might be a fix for this so it can't happen again. A virtual host can be limited in how many connections they allow. This way Windows won't faulter on the many file handles. Look in the management portal -> Admin -> Limits, set max-connections to a number you think it plausible for your setup.

Answer 4

I opted to recreate the vhost which resolved the issue for me.

Clearing the data in transient msg_stores & queues didn't work for me in my scenario. Since data loss wasn't an issue for me.

Answer 5

For me, I was trying to install management plugin (rabbitmqadmin) and it failed in rabbitmq-delayed-message-exchange. So I downloaded rabbitmq-delayed-message-exchange from JFrog Bintray and updated my plugins folder, stopped and started rabbitmq service. This action, courrupted the database.

To fix it,

1. Stop Rabbitmq service

2. I went to %APPDATA%\RabbitMQ\db and cleared the data in transient msg_stores & queues folders. e.g. C:\Users\bharath\AppData\Roaming\RabbitMQ\db\rabbit@js-mnesia\msg_stores\vhosts\6ZOILVXV632BBEWZY4TBABTPP\msg_store_transient

3. Removed the rabbitmq-delayed-message-exchange from rabbit@js-plugins-expand folder

4. Removed the data in transient msg_store & queues from RABBIT~1-upgrade-backup folder

5. Went one folder back and removed the plugin name rabbitmq-delayed-message-exchange, from enabled_plugins file

6. Start the rabbitmq service

It worked!!!

Note: Only try this if data is not important, as it will get cleared or you can restore it from backup.