RabbitMQ queues crash with an Exception in queue index recovery

Products

• VMware RabbitMQ

Issue/Introduction

RabbitMQ queues crash randomly with the following errors:
 

[error] <0.6083.0> ** Generic server <0.6083.0> terminating
[error] <0.6083.0> ** Last message in was {basic_get,<0.9595.36>,true,<0.24603.35>}
[error] <0.6083.0> ** When Server state == {q,
[error] <0.6083.0>                          {amqqueue,
[error] <0.6083.0>                           {resource,<<"xxxxx">>,queue,
[error] <0.6083.0>                            <<"FormatName:DIRECT=OS:TSD-EQ-P-AP02\\private$\\ws_ETON_out">>},
[error] <0.6083.0>                           true,false,none,[],<0.6083.0>,[],[],[],undefined,
[error] <0.6083.0>                           undefined,[],undefined,stopped,0,[],<<"xxxxx">>,
[error] <0.6083.0>                           #{user => <<"userY">>},
[error] <0.6083.0>                           rabbit_classic_queue,#{}},
[error] <0.6083.0>                          none,false,rabbit_priority_queue,
[error] <0.6083.0>                          {passthrough,rabbit_variable_queue,
[error] <0.6083.0>                           {vqstate,
[error] <0.6083.0>                            {97,
...
[error] <0.6083.0> ** Reason for termination ==
[error] <0.6083.0> ** {function_clause,
[error] <0.6083.0>        [{rabbit_queue_index,segment_plus_journal1,
[error] <0.6083.0>             [{{true,
[error] <0.6083.0>                   <<39,173,136,100,84,175,187,31,92,217,65,184,116,168,119,230,
...
[error] <0.6083.0>   crasher:
[error] <0.6083.0>     initial call: rabbit_prequeue:init/1
[error] <0.6083.0>     pid: <0.6083.0>
[error] <0.6083.0>     registered_name: []
[error] <0.6083.0>     exception exit: {function_clause,
[error] <0.6083.0>                         [{rabbit_queue_index,segment_plus_journal1,
[error] <0.6083.0>                              [{{true,
[error] <0.9595.36>                                      <<39,173,136,100,84,175,187,31,92,217,65,
[error] <0.9595.36>                                        184,116,168,119,230,0,0,0,0,0,0,0,0,0,0,
...
[error] <0.9595.36>               [{file,"rabbit_queue_index.erl"},{line,1347}]},
[error] <0.9595.36>           {rabbit_queue_index,'-segment_plus_journal/2-fun-0-',3,
[error] <0.9595.36>               [{file,"rabbit_queue_index.erl"},{line,1334}]},
[error] <0.9595.36>           {array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},
[error] <0.9595.36>           {array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},
[error] <0.9595.36>           {rabbit_queue_index,segment_entries_foldr,3,
[error] <0.9595.36>               [{file,"rabbit_queue_index.erl"},{line,1240}]},
[error] <0.9595.36>           {rabbit_queue_index,read_bounded_segment,5,
[error] <0.9595.36>               [{file,"rabbit_queue_index.erl"},{line,1221}]}

The corresponding Application logs show the following error message:

The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=541, text='INTERNAL_ERROR', classId=0, methodId=0

Resolution

This is a known product issue. RabbitMQ versions 3.12 and newer include CQv2, which does not exhibit this behavior. For this reason, this issue can be resolved by upgrading to RabbitMQ 3.12 or higher and switching to CQv2. 


References:

RabbitMQ 3.12 Performance Improvements
Exception in queue index recovery #7203
The RabbitMQ classic queue Storage Implementation Versions
RabbitMQ 3.12 release notes