RabbitMQ cluster for TAS fails to restart with '[error] <0.xxx.0> rabbit_queue_index,journal_minus_segment1'
Article ID: 293257
Updated On:
Products
VMware RabbitMQ
Issue/Introduction
RabbitMQ cluster fails to restart with the error shown below generated during the queue index recovery process.
2023-05-16 16:03:02.548647+00:00 [error] <0.364.0> {array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]}, 2023-05-16 16:03:02.548647+00:00 [error] <0.364.0> {rabbit_queue_index,'-recover_journal/1-fun-0-',1, 2023-05-16 16:03:02.548647+00:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1059}]}, 2023-05-16 16:03:02.548647+00:00 [error] <0.364.0> {lists,map,2,[{file,"lists.erl"},{line,1315}]}, 2023-05-16 16:03:02.548647+00:00 [error] <0.364.0> {rabbit_queue_index,segment_map,2, 2023-05-16 16:03:02.548647+00:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1183}]}, 2023-05-16 16:03:02.548647+00:00 [error] <0.364.0> {rabbit_queue_index,recover_journal,1, 2023-05-16 16:03:02.548647+00:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1050}]}]} 2023-05-16 16:03:02.548647+00:00 [error] <0.364.0> 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> crasher: 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> initial call: worker_pool_worker:init/1 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> pid: <0.364.0> 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> registered_name: [] 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> exception exit: {function_clause, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> [{rabbit_queue_index,journal_minus_segment1, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> [{no_pub,no_del,ack}, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> {{true, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> <<157,228,174,158,48,127,102,9,226,142,174, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> 34,84,51,203,79,0,5,251,199,85,5,64,98,0, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> 0,0,201>>, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> <<>>}, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> no_del,no_ack}], 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1395}]}, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> {rabbit_queue_index, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> '-journal_minus_segment/3-fun-0-',4, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1372}]}, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> {array,sparse_foldl_3,7, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> [{file,"array.erl"},{line,1684}]}, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> {array,sparse_foldl_2,9, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> [{file,"array.erl"},{line,1678}]}, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> {rabbit_queue_index,'-recover_journal/1-fun-0-',1, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1059}]}, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> {lists,map,2,[{file,"lists.erl"},{line,1315}]}, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> {rabbit_queue_index,segment_map,2, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1183}]}, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> {rabbit_queue_index,recover_journal,1, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> [{file,"rabbit_queue_index.erl"},{line,1050}]}]} 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1183) 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> ancestors: [worker_pool_sup,rabbit_sup,<0.223.0>] 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> message_queue_len: 0 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> messages: [] 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> links: [<0.361.0>] 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> dictionary: [{segment_entry_count,16384}, 2023-05-16 16:03:02.554127+00:00 [error] <0.364.0> {{"/var/vcap/store/rabbitmq/mnesia/db/msg_stores/vhosts/AvHost/queues/AQueue/journal.jif",
From the logs we can see that queue "AQueue" on vhost "AvHost" has corrupted data that is preventing the node from coming up.
Environment
Product Version: Other
Resolution
For RabbitMQ for TAS follow the steps below to recover the cluster.
1.Take a backup of only the affected queue folder (/var/vcap/store/rabbitmq/mnesia/db/msg_stores/vhosts/<vhost_name>/queues/<queue_name)
2. Run monit stop rabbitmq-server
3. Run monit start rabbitmq-server
Note that the steps listed above are for the TAS environment, but a similar situation may occur in OSS RabbitMQ where the process is similar but the exact steps would be different.
1.Take a backup of only the affected queue folder (/var/vcap/store/rabbitmq/mnesia/db/msg_stores/vhosts/<vhost_name>/queues/<queue_name)
2. Run monit stop rabbitmq-server
3. Run monit start rabbitmq-server
Note that the steps listed above are for the TAS environment, but a similar situation may occur in OSS RabbitMQ where the process is similar but the exact steps would be different.
Feedback
Yes
No
Powered by
